2026-10-29 19:53:56,841 [ 286232 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2026-10-29 19:53:56,841 [ 286232 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2026-10-29 19:53:56,841 [ 286232 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2026-10-29 19:53:56,841 [ 286232 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_sbkw4t --privileged --dns-search='.' --memory=30703235072 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel2_0.jsonl --report-log-exclude-logs-on-passed-tests 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1]' test_recompression_ttl/test.py::test_recompression_multiple_ttls test_recompression_ttl/test.py::test_recompression_replicated test_recompression_ttl/test.py::test_recompression_simple test_refreshable_mv/test.py::test_adding_replica test_refreshable_mv/test.py::test_backup_inner_table test_refreshable_mv/test.py::test_backup_outer_table test_refreshable_mv/test.py::test_pause test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_refreshable_mv/test.py::test_refreshable_mv_in_read_only_node test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db test_refreshable_mv/test.py::test_refreshable_mv_in_system_db test_remove_stale_moving_parts/test.py::test_remove_stale_moving_parts_without_zookeeper test_render_log_file_name_templates/test.py::test_check_file_names test_replicated_database_alter_modify_order_by/test.py::test_alter_modify_order_by test_replicated_engine_arguments/test.py::test_replicated_engine_with_arguments test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table 'test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16]' 'test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12]' test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore test_replicated_user_defined_functions/test.py::test_create_and_drop 'test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[false-expected_raise1]' 'test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[true-expected_raise0]' test_replicated_user_defined_functions/test.py::test_create_and_replace test_replicated_user_defined_functions/test.py::test_drop_if_exists test_replicated_user_defined_functions/test.py::test_reload_zookeeper test_replicated_user_defined_functions/test.py::test_replication test_replicated_user_defined_functions/test.py::test_replication_replace_by_another_node_after_creation test_replicated_user_defined_functions/test.py::test_server_restart test_replicated_user_defined_functions/test.py::test_start_without_zookeeper test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 'test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header]' test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster test_s3_zero_copy_ttl/test.py::test_ttl_move_and_s3 test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage test_secure_socket/test.py::test 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6]' test_settings_constraints_distributed/test.py::test_insert_clamps_settings test_settings_constraints_distributed/test.py::test_select_clamps_settings test_settings_randomization/test.py::test_settings_randomization test_shard_level_const_function/test.py::test_remote test_shard_names/test.py::test_incorrect_shard_names test_shard_names/test.py::test_shard_names test_shutdown_static_destructor_failure/test.py::test_shutdown test_ssh/test.py::test_create_table test_ssh/test.py::test_no_inserts_from_infile test_ssh/test.py::test_no_queries_from_file test_ssh/test.py::test_no_selects_into_outfile test_ssh/test.py::test_simple_query_with_openssh_client test_ssh/test.py::test_simple_query_with_paramiko test_storage_delta/test_imds.py::test_session_token 'test_storage_iceberg/test.py::test_cluster_table_function[azure-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[azure-2]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-2]' 'test_storage_iceberg/test.py::test_cluster_table_function_with_partition_pruning[s3-2]' 'test_storage_iceberg/test.py::test_delete_files[azure-1]' 'test_storage_iceberg/test.py::test_delete_files[azure-2]' 'test_storage_iceberg/test.py::test_delete_files[local-1]' 'test_storage_iceberg/test.py::test_delete_files[local-2]' 'test_storage_iceberg/test.py::test_delete_files[s3-1]' 'test_storage_iceberg/test.py::test_delete_files[s3-2]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[azure-1]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[azure-2]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[local-1]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[local-2]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[s3-1]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[s3-2]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-1]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-2]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-1]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-2]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-1]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-2]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-1]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-2]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_ssh/test.py::test_create_table test_storage_iceberg/test.py::test_cluster_table_function[azure-1] test_replicated_user_defined_functions/test.py::test_create_and_drop test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1] test_s3_cluster/test.py::test_ambiguous_join test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] test_recompression_ttl/test.py::test_recompression_multiple_ttls test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys test_refreshable_mv/test.py::test_adding_replica [gw5] [ 1%] PASSED test_ssh/test.py::test_create_table test_ssh/test.py::test_no_inserts_from_infile [gw5] [ 2%] PASSED test_ssh/test.py::test_no_inserts_from_infile test_ssh/test.py::test_no_queries_from_file [gw5] [ 3%] PASSED test_ssh/test.py::test_no_queries_from_file test_ssh/test.py::test_no_selects_into_outfile [gw5] [ 4%] PASSED test_ssh/test.py::test_no_selects_into_outfile test_ssh/test.py::test_simple_query_with_openssh_client [gw5] [ 5%] PASSED test_ssh/test.py::test_simple_query_with_openssh_client test_ssh/test.py::test_simple_query_with_paramiko [gw5] [ 6%] PASSED test_ssh/test.py::test_simple_query_with_paramiko test_shard_names/test.py::test_incorrect_shard_names [gw8] [ 7%] PASSED test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] [gw1] [ 8%] PASSED test_replicated_user_defined_functions/test.py::test_create_and_drop test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[false-expected_raise1] [gw1] [ 9%] PASSED test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[false-expected_raise1] test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[true-expected_raise0] [gw8] [ 10%] PASSED test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] [gw3] [ 11%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1] test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2] [gw1] [ 12%] PASSED test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[true-expected_raise0] test_replicated_user_defined_functions/test.py::test_create_and_replace [gw2] [ 13%] PASSED test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression [gw1] [ 14%] PASSED test_replicated_user_defined_functions/test.py::test_create_and_replace test_replicated_user_defined_functions/test.py::test_drop_if_exists [gw8] [ 15%] PASSED test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] [gw1] [ 16%] PASSED test_replicated_user_defined_functions/test.py::test_drop_if_exists test_replicated_user_defined_functions/test.py::test_reload_zookeeper [gw3] [ 17%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2] test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3] [gw4] [ 18%] PASSED test_refreshable_mv/test.py::test_adding_replica test_refreshable_mv/test.py::test_backup_inner_table [gw9] [ 19%] PASSED test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys [gw2] [ 20%] PASSED test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection [gw3] [ 21%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3] test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4] [gw5] [ 22%] PASSED test_shard_names/test.py::test_incorrect_shard_names test_shard_names/test.py::test_shard_names [gw2] [ 23%] PASSED test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header [gw5] [ 24%] PASSED test_shard_names/test.py::test_shard_names [gw3] [ 25%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4] test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5] [gw9] [ 26%] FAILED test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys [gw3] [ 27%] FAILED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5] test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6] [gw2] [ 28%] PASSED test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_remove_stale_moving_parts/test.py::test_remove_stale_moving_parts_without_zookeeper [gw2] [ 29%] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count [gw2] [ 30%] PASSED test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro [gw0] [ 31%] PASSED test_storage_iceberg/test.py::test_cluster_table_function[azure-1] test_storage_iceberg/test.py::test_cluster_table_function[azure-2] test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] [gw3] [ 32%] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6] [gw2] [ 33%] PASSED test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated [gw2] [ 34%] FAILED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference [gw2] [ 35%] FAILED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference test_s3_cluster/test.py::test_select_all [gw2] [ 36%] PASSED test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards [gw2] [ 37%] PASSED test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all [gw6] [ 38%] PASSED test_recompression_ttl/test.py::test_recompression_multiple_ttls test_recompression_ttl/test.py::test_recompression_replicated test_settings_constraints_distributed/test.py::test_insert_clamps_settings [gw2] [ 39%] PASSED test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards [gw2] [ 40%] PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster [gw2] [ 41%] PASSED test_s3_cluster/test.py::test_wrong_cluster test_replicated_engine_arguments/test.py::test_replicated_engine_with_arguments [gw0] [ 42%] PASSED test_storage_iceberg/test.py::test_cluster_table_function[azure-2] test_storage_iceberg/test.py::test_cluster_table_function[s3-1] [gw4] [ 43%] FAILED test_refreshable_mv/test.py::test_backup_inner_table [gw6] [ 44%] PASSED test_recompression_ttl/test.py::test_recompression_replicated test_recompression_ttl/test.py::test_recompression_simple [gw0] [ 45%] PASSED test_storage_iceberg/test.py::test_cluster_table_function[s3-1] test_storage_iceberg/test.py::test_cluster_table_function[s3-2] test_secure_socket/test.py::test [gw1] [ 46%] PASSED test_replicated_user_defined_functions/test.py::test_reload_zookeeper test_replicated_user_defined_functions/test.py::test_replication [gw5] [ 47%] FAILED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] [gw3] [ 48%] PASSED test_replicated_engine_arguments/test.py::test_replicated_engine_with_arguments test_refreshable_mv/test.py::test_backup_outer_table [gw8] [ 49%] PASSED test_remove_stale_moving_parts/test.py::test_remove_stale_moving_parts_without_zookeeper [gw0] [ 50%] PASSED test_storage_iceberg/test.py::test_cluster_table_function[s3-2] test_replicated_database_alter_modify_order_by/test.py::test_alter_modify_order_by test_storage_iceberg/test.py::test_cluster_table_function_with_partition_pruning[s3-2] [gw1] [ 51%] PASSED test_replicated_user_defined_functions/test.py::test_replication test_replicated_user_defined_functions/test.py::test_replication_replace_by_another_node_after_creation test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility [gw9] [ 52%] PASSED test_settings_constraints_distributed/test.py::test_insert_clamps_settings test_settings_constraints_distributed/test.py::test_select_clamps_settings [gw0] [ 53%] PASSED test_storage_iceberg/test.py::test_cluster_table_function_with_partition_pruning[s3-2] test_storage_iceberg/test.py::test_delete_files[azure-1] [gw1] [ 54%] PASSED test_replicated_user_defined_functions/test.py::test_replication_replace_by_another_node_after_creation test_replicated_user_defined_functions/test.py::test_server_restart [gw2] [ 55%] FAILED test_secure_socket/test.py::test [gw6] [ 56%] PASSED test_recompression_ttl/test.py::test_recompression_simple test_render_log_file_name_templates/test.py::test_check_file_names [gw4] [ 57%] FAILED test_refreshable_mv/test.py::test_backup_outer_table [gw0] [ 58%] PASSED test_storage_iceberg/test.py::test_delete_files[azure-1] test_storage_iceberg/test.py::test_delete_files[azure-2] test_refreshable_mv/test.py::test_pause [gw4] [ 59%] FAILED test_refreshable_mv/test.py::test_pause test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_settings_randomization/test.py::test_settings_randomization [gw8] [ 60%] PASSED test_render_log_file_name_templates/test.py::test_check_file_names [gw0] [ 61%] PASSED test_storage_iceberg/test.py::test_delete_files[azure-2] test_storage_iceberg/test.py::test_delete_files[local-1] [gw1] [ 62%] PASSED test_replicated_user_defined_functions/test.py::test_server_restart test_replicated_user_defined_functions/test.py::test_start_without_zookeeper test_s3_zero_copy_ttl/test.py::test_ttl_move_and_s3 [gw5] [ 63%] PASSED test_replicated_database_alter_modify_order_by/test.py::test_alter_modify_order_by [gw9] [ 64%] PASSED test_settings_constraints_distributed/test.py::test_select_clamps_settings test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss [gw2] [ 65%] PASSED test_settings_randomization/test.py::test_settings_randomization test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage [gw0] [ 66%] PASSED test_storage_iceberg/test.py::test_delete_files[local-1] test_storage_iceberg/test.py::test_delete_files[local-2] [gw7] [ 67%] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table test_storage_delta/test_imds.py::test_session_token test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] [gw7] [ 68%] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] [gw0] [ 69%] PASSED test_storage_iceberg/test.py::test_delete_files[local-2] test_storage_iceberg/test.py::test_delete_files[s3-1] [gw9] [ 70%] SKIPPED test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage [gw7] [ 71%] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] [gw0] [ 72%] PASSED test_storage_iceberg/test.py::test_delete_files[s3-1] test_storage_iceberg/test.py::test_delete_files[s3-2] [gw8] [ 73%] PASSED test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore [gw0] [ 74%] PASSED test_storage_iceberg/test.py::test_delete_files[s3-2] test_storage_iceberg/test.py::test_evolved_schema_complex[azure-1] [gw7] [ 75%] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore [gw0] [ 76%] PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[azure-1] test_storage_iceberg/test.py::test_evolved_schema_complex[azure-2] [gw3] [ 77%] PASSED test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility [gw0] [ 78%] PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[azure-2] test_storage_iceberg/test.py::test_evolved_schema_complex[local-1] [gw6] [ 79%] PASSED test_s3_zero_copy_ttl/test.py::test_ttl_move_and_s3 [gw0] [ 80%] PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[local-1] test_storage_iceberg/test.py::test_evolved_schema_complex[local-2] [gw1] [ 81%] PASSED test_replicated_user_defined_functions/test.py::test_start_without_zookeeper [gw0] [ 82%] PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[local-2] test_storage_iceberg/test.py::test_evolved_schema_complex[s3-1] [gw2] [ 83%] PASSED test_storage_delta/test_imds.py::test_session_token [gw0] [ 84%] PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[s3-1] test_storage_iceberg/test.py::test_evolved_schema_complex[s3-2] [gw0] [ 85%] PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[s3-2] test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-1] test_shard_level_const_function/test.py::test_remote test_shutdown_static_destructor_failure/test.py::test_shutdown [gw0] [ 86%] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-1] test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-2] [gw6] [ 87%] PASSED test_shutdown_static_destructor_failure/test.py::test_shutdown [gw3] [ 88%] PASSED test_shard_level_const_function/test.py::test_remote [gw0] [ 89%] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-2] test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-1] [gw0] [ 90%] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-1] test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-2] [gw4] [ 91%] PASSED test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_refreshable_mv/test.py::test_refreshable_mv_in_read_only_node [gw0] [ 92%] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-2] test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-1] [gw4] [ 93%] PASSED test_refreshable_mv/test.py::test_refreshable_mv_in_read_only_node test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db [gw0] [ 94%] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-1] test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-2] [gw0] [ 95%] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-2] test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-1] [gw0] [ 96%] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-1] test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-2] [gw0] [ 97%] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-2] [gw4] [ 98%] PASSED test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db test_refreshable_mv/test.py::test_refreshable_mv_in_system_db [gw4] [ 99%] PASSED test_refreshable_mv/test.py::test_refreshable_mv_in_system_db [gw5] [100%] PASSED test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss =================================== FAILURES =================================== ________________________________ test_same_keys ________________________________ [gw9] linux -- Python 3.10.12 /usr/bin/python3 def test_same_keys(): copy_keys(node1, "key_a") copy_keys(node2, "key_a") create_table() insert_data() check_table() optimize_table() > check_table() test_replicated_merge_tree_encrypted_disk/test.py:91: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def check_table(): expected = [[1, "str1"], [2, "str2"]] node1.query("SYSTEM SYNC REPLICA tbl LIGHTWEIGHT") node2.query("SYSTEM SYNC REPLICA tbl LIGHTWEIGHT") assert node1.query("SELECT * FROM tbl ORDER BY id") == TSV(expected) assert node2.query("SELECT * FROM tbl ORDER BY id") == TSV(expected) > assert node1.query("CHECK TABLE tbl") == "1\n" E AssertionError: assert '0\n' == '1\n' E - 1 E + 0 test_replicated_merge_tree_encrypted_disk/test.py:75: AssertionError ----------------------------- Captured stderr call ----------------------------- run container_id:roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml'] run container_id:roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml'] Command:[docker exec roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml] Command:[docker exec roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml] Executing query SYSTEM RELOAD CONFIG on node1 Executing query SYSTEM RELOAD CONFIG on node1 run container_id:roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml'] run container_id:roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml'] Command:[docker exec roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml] Command:[docker exec roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml] Executing query SYSTEM RELOAD CONFIG on node2 Executing query SYSTEM RELOAD CONFIG on node2 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster' SYNC on node1 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster' SYNC on node1 Executing query CREATE TABLE tbl ON CLUSTER 'cluster' ( id Int64, str String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}') ORDER BY id SETTINGS storage_policy='encrypted_policy' on node1 Executing query CREATE TABLE tbl ON CLUSTER 'cluster' ( id Int64, str String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}') ORDER BY id SETTINGS storage_policy='encrypted_policy' on node1 Executing query INSERT INTO tbl VALUES (1, 'str1') on node1 Executing query INSERT INTO tbl VALUES (1, 'str1') on node1 Executing query INSERT INTO tbl VALUES (1, 'str1') on node2 Executing query INSERT INTO tbl VALUES (1, 'str1') on node2 Executing query INSERT INTO tbl VALUES (2, 'str2') on node2 Executing query INSERT INTO tbl VALUES (2, 'str2') on node2 Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node1 Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node1 Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node2 Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node2 Executing query SELECT * FROM tbl ORDER BY id on node1 Executing query SELECT * FROM tbl ORDER BY id on node1 Executing query SELECT * FROM tbl ORDER BY id on node2 Executing query SELECT * FROM tbl ORDER BY id on node2 Executing query CHECK TABLE tbl on node1 Executing query CHECK TABLE tbl on node1 Executing query CHECK TABLE tbl on node2 Executing query CHECK TABLE tbl on node2 Executing query OPTIMIZE TABLE tbl ON CLUSTER 'cluster' FINAL on node1 Executing query OPTIMIZE TABLE tbl ON CLUSTER 'cluster' FINAL on node1 Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node1 Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node1 Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node2 Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node2 Executing query SELECT * FROM tbl ORDER BY id on node1 Executing query SELECT * FROM tbl ORDER BY id on node1 Executing query SELECT * FROM tbl ORDER BY id on node2 Executing query SELECT * FROM tbl ORDER BY id on node2 Executing query CHECK TABLE tbl on node1 Executing query CHECK TABLE tbl on node1 ------------------------------ Captured log call ------------------------------- 2026-10-29 19:54:36.561000 [ 762 ] DEBUG : run container_id:roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml'] (cluster.py:2051, exec_in_container) 2026-10-29 19:54:36.561000 [ 762 ] DEBUG : Command:[docker exec roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml] (cluster.py:121, run_and_check) 2026-10-29 19:54:36.827000 [ 762 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node1 (cluster.py:3648, query) 2026-10-29 19:54:37.420000 [ 762 ] DEBUG : run container_id:roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml'] (cluster.py:2051, exec_in_container) 2026-10-29 19:54:37.421000 [ 762 ] DEBUG : Command:[docker exec roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/config.d/z_keys.xml) && echo PGNsaWNraG91c2U+CiAgICA8c3RvcmFnZV9jb25maWd1cmF0aW9uPgogICAgICAgIDxkaXNrcz4KICAgICAgICAgICAgPGRpc2tfZW5jcnlwdGVkPgogICAgICAgICAgICAgICAgPGtleT5hYWFhYWFhYWFhYWFhYWFhPC9rZXk+CiAgICAgICAgICAgIDwvZGlza19lbmNyeXB0ZWQ+CiAgICAgICAgPC9kaXNrcz4KICAgIDwvc3RvcmFnZV9jb25maWd1cmF0aW9uPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/z_keys.xml] (cluster.py:121, run_and_check) 2026-10-29 19:54:37.530000 [ 762 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node2 (cluster.py:3648, query) 2026-10-29 19:54:38.123000 [ 762 ] DEBUG : Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster' SYNC on node1 (cluster.py:3648, query) 2026-10-29 19:54:38.863000 [ 762 ] DEBUG : Executing query CREATE TABLE tbl ON CLUSTER 'cluster' ( id Int64, str String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}') ORDER BY id SETTINGS storage_policy='encrypted_policy' on node1 (cluster.py:3648, query) 2026-10-29 19:54:40.023000 [ 762 ] DEBUG : Executing query INSERT INTO tbl VALUES (1, 'str1') on node1 (cluster.py:3648, query) 2026-10-29 19:54:40.394000 [ 762 ] DEBUG : Executing query INSERT INTO tbl VALUES (1, 'str1') on node2 (cluster.py:3648, query) 2026-10-29 19:54:43.057000 [ 762 ] DEBUG : Executing query INSERT INTO tbl VALUES (2, 'str2') on node2 (cluster.py:3648, query) 2026-10-29 19:54:43.567000 [ 762 ] DEBUG : Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node1 (cluster.py:3648, query) 2026-10-29 19:54:44.137000 [ 762 ] DEBUG : Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node2 (cluster.py:3648, query) 2026-10-29 19:54:44.459000 [ 762 ] DEBUG : Executing query SELECT * FROM tbl ORDER BY id on node1 (cluster.py:3648, query) 2026-10-29 19:54:45.062000 [ 762 ] DEBUG : Executing query SELECT * FROM tbl ORDER BY id on node2 (cluster.py:3648, query) 2026-10-29 19:54:45.621000 [ 762 ] DEBUG : Executing query CHECK TABLE tbl on node1 (cluster.py:3648, query) 2026-10-29 19:54:46.030000 [ 762 ] DEBUG : Executing query CHECK TABLE tbl on node2 (cluster.py:3648, query) 2026-10-29 19:54:46.663000 [ 762 ] DEBUG : Executing query OPTIMIZE TABLE tbl ON CLUSTER 'cluster' FINAL on node1 (cluster.py:3648, query) 2026-10-29 19:54:49.244000 [ 762 ] DEBUG : Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node1 (cluster.py:3648, query) 2026-10-29 19:54:49.799000 [ 762 ] DEBUG : Executing query SYSTEM SYNC REPLICA tbl LIGHTWEIGHT on node2 (cluster.py:3648, query) 2026-10-29 19:54:53.899000 [ 762 ] DEBUG : Executing query SELECT * FROM tbl ORDER BY id on node1 (cluster.py:3648, query) 2026-10-29 19:54:54.585000 [ 762 ] DEBUG : Executing query SELECT * FROM tbl ORDER BY id on node2 (cluster.py:3648, query) 2026-10-29 19:54:55.313000 [ 762 ] DEBUG : Executing query CHECK TABLE tbl on node1 (cluster.py:3648, query) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/.env --project-name roottestreplicatedmergetreeencrypteddisk-gw9 --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/docker-compose.yml stop --timeout 20] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/.env --project-name roottestreplicatedmergetreeencrypteddisk-gw9 --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/docker-compose.yml stop --timeout 20] Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/.env --project-name roottestreplicatedmergetreeencrypteddisk-gw9 --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/docker-compose.yml down --volumes] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/.env --project-name roottestreplicatedmergetreeencrypteddisk-gw9 --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/docker-compose.yml down --volumes] Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Removing Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Removing Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Removing Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Removing Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Removed Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Removed Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Removed Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Removed Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopping Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Removing Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Removing Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Removing Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Removing Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopped Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Removing Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Removing Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Removed Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Removed Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Removed Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Removed Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Removed Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Removed Stderr: Network roottestreplicatedmergetreeencrypteddisk-gw9_default Removing Stderr: Network roottestreplicatedmergetreeencrypteddisk-gw9_default Removing Stderr: Network roottestreplicatedmergetreeencrypteddisk-gw9_default Removed Stderr: Network roottestreplicatedmergetreeencrypteddisk-gw9_default Removed Cleanup called Cleanup called Docker networks for project roottestreplicatedmergetreeencrypteddisk-gw9 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestreplicatedmergetreeencrypteddisk-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicatedmergetreeencrypteddisk-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestreplicatedmergetreeencrypteddisk-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicatedmergetreeencrypteddisk-gw9 are DRIVER VOLUME NAME Docker volumes for project roottestreplicatedmergetreeencrypteddisk-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicatedmergetreeencrypteddisk-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestreplicatedmergetreeencrypteddisk-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestreplicatedmergetreeencrypteddisk-gw9 No running containers for project: roottestreplicatedmergetreeencrypteddisk-gw9 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:10 Stdout:10 Command:[docker volume prune -f] Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Volumes pruned: 10 Volumes pruned: 10 ---------------------------- Captured log teardown ----------------------------- 2026-10-29 19:54:56.722000 [ 762 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/.env --project-name roottestreplicatedmergetreeencrypteddisk-gw9 --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/docker-compose.yml stop --timeout 20] (cluster.py:121, run_and_check) 2026-10-29 19:55:31.103000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:55:31.135000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:55:31.135000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:55:31.136000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:55:31.136000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:55:31.136000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:55:31.136000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:55:31.136000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:55:31.136000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:55:31.136000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:55:31.144000 [ 762 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:55:31.279000 [ 762 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:55:31.367000 [ 762 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/.env --project-name roottestreplicatedmergetreeencrypteddisk-gw9 --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicated_merge_tree_encrypted_disk/_instances-0-gw9/node2/docker-compose.yml down --volumes] (cluster.py:121, run_and_check) 2026-10-29 19:55:34.451000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:55:34.451000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:55:34.451000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:55:34.451000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:55:34.451000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:55:34.451000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:55:34.451000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node2-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:55:34.451000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-node1-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:55:34.451000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo2-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo1-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Container roottestreplicatedmergetreeencrypteddisk-gw9-zoo3-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Network roottestreplicatedmergetreeencrypteddisk-gw9_default Removing (cluster.py:147, run_and_check) 2026-10-29 19:55:34.452000 [ 762 ] DEBUG : Stderr: Network roottestreplicatedmergetreeencrypteddisk-gw9_default Removed (cluster.py:147, run_and_check) 2026-10-29 19:55:34.453000 [ 762 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 19:55:34.482000 [ 762 ] DEBUG : Docker networks for project roottestreplicatedmergetreeencrypteddisk-gw9 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 19:55:34.527000 [ 762 ] DEBUG : Docker containers for project roottestreplicatedmergetreeencrypteddisk-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 19:55:34.591000 [ 762 ] DEBUG : Docker volumes for project roottestreplicatedmergetreeencrypteddisk-gw9 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 19:55:34.591000 [ 762 ] DEBUG : Command:[docker container list --all --filter name='^/roottestreplicatedmergetreeencrypteddisk-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 19:55:34.705000 [ 762 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 19:55:34.705000 [ 762 ] DEBUG : No running containers for project: roottestreplicatedmergetreeencrypteddisk-gw9 (cluster.py:879, cleanup) 2026-10-29 19:55:34.705000 [ 762 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 19:55:34.740000 [ 762 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 19:55:34.740000 [ 762 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:55:34.808000 [ 762 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:55:34.810000 [ 762 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 19:55:34.810000 [ 762 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 19:55:34.810000 [ 762 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 19:55:34.904000 [ 762 ] DEBUG : Stdout:10 (cluster.py:145, run_and_check) 2026-10-29 19:55:34.904000 [ 762 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:55:34.979000 [ 762 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:55:34.979000 [ 762 ] DEBUG : Volumes pruned: 10 (cluster.py:915, cleanup) ____________________ test_alter_table_drop_partition[case5] ____________________ [gw3] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = table = 'someothertable', query = 'TRUNCATE TABLE someothertable' expected = '0' n1 = n2 = @pytest.mark.parametrize( "table,query,expected,n1,n2", [ pytest.param( "sometable", "ALTER TABLE sometable DROP PARTITION 201706", "1", node1, node2, id="case1", ), pytest.param( "sometable", "TRUNCATE TABLE sometable", "0", node1, node2, id="case2" ), pytest.param( "sometable", "OPTIMIZE TABLE sometable", "4", node1, node2, id="case3" ), pytest.param( "someothertable", "ALTER TABLE someothertable DROP PARTITION 201706", "1", node3, node4, id="case4", ), pytest.param( "someothertable", "TRUNCATE TABLE someothertable", "0", node3, node4, id="case5", ), pytest.param( "someothertable", "OPTIMIZE TABLE someothertable", "4", node3, node4, id="case6", ), ], ) def test_alter_table_drop_partition(started_cluster, table, query, expected, n1, n2): to_insert = """\ 2017-06-16 111 0 2017-06-16 222 1 2017-06-16 333 2 2017-07-16 444 3 """ n1.query("INSERT INTO {} FORMAT TSV".format(table), stdin=to_insert, user="good") assert_eq_with_retry(n1, "SELECT COUNT(*) from {}".format(table), "4", user="good") assert_eq_with_retry(n2, "SELECT COUNT(*) from {}".format(table), "4", user="good") ### It maybe leader and everything will be ok > n1.query(query, user="good") test_send_request_to_leader_replica/test.py:123: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3649: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 231, stderr: Received exception from server (version 25.3.14): E Code: 999. DB::Exception: Received from 172.16.1.7:9000. Coordination::Exception. Coordination::Exception: Coordination error: Connection loss, path /clickhouse/tables/0/someothertable/block_numbers. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb E 3. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type, std::type_identity::type>, char const*&&, String const&) @ 0x0000000010ef6204 E 4. Coordination::Exception::fromPath(Coordination::Error, String const&) @ 0x0000000010ef541f E 5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:378: zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000315ecb48 E 6. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:348: zkutil::ZooKeeper::tryGetChildren(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr const&, Coordination::ListRequestType) @ 0x00000000315ec1fe E 7. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:323: zkutil::ZooKeeper::getChildren(String const&, Coordination::Stat*, std::shared_ptr const&, Coordination::ListRequestType) @ 0x00000000315ebf6c E 8. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:6881: DB::StorageReplicatedMergeTree::dropAllPartitionsImpl(std::shared_ptr const&, bool, std::shared_ptr) @ 0x000000002e9522b7 E 9. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:6943: DB::StorageReplicatedMergeTree::truncate(std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr, DB::TableExclusiveLockHolder&) @ 0x000000002e956837 E 10. ./build_docker/./src/Interpreters/InterpreterDropQuery.cpp:282: DB::InterpreterDropQuery::executeToTableImpl(std::shared_ptr const&, DB::ASTDropQuery&, std::shared_ptr&, StrongTypedef, DB::UUIDTag>&) @ 0x000000002a6bf497 E 11. ./build_docker/./src/Interpreters/InterpreterDropQuery.cpp:118: DB::InterpreterDropQuery::executeToTable(DB::ASTDropQuery&) @ 0x000000002a6bb6bd E 12. ./build_docker/./src/Interpreters/InterpreterDropQuery.cpp:91: DB::InterpreterDropQuery::executeSingleDropQuery(std::shared_ptr const&) @ 0x000000002a6b9d41 E 13. ./build_docker/./src/Interpreters/InterpreterDropQuery.cpp:72: DB::InterpreterDropQuery::execute() @ 0x000000002a6b97cd E 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1457: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af36de3 E 15. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 E 16. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 E 17. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa E 18. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f E 19. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 E 20. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b E 21. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 E 22. asan_thread_start(void*) @ 0x000000000c676e37 E 23. ? @ 0x00007f0966916ac3 E 24. ? @ 0x00007f09669a8850 E . (KEEPER_EXCEPTION) E (query: TRUNCATE TABLE someothertable) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query INSERT INTO someothertable FORMAT TSV on node3 Executing query INSERT INTO someothertable FORMAT TSV on node3 Executing query SELECT COUNT(*) from someothertable on node3 Executing query SELECT COUNT(*) from someothertable on node3 Executing query SELECT COUNT(*) from someothertable on node4 Executing query SELECT COUNT(*) from someothertable on node4 Executing query TRUNCATE TABLE someothertable on node3 Executing query TRUNCATE TABLE someothertable on node3 ------------------------------ Captured log call ------------------------------- 2026-10-29 19:54:49.410000 [ 684 ] DEBUG : Executing query INSERT INTO someothertable FORMAT TSV on node3 (cluster.py:3648, query) 2026-10-29 19:54:49.916000 [ 684 ] DEBUG : Executing query SELECT COUNT(*) from someothertable on node3 (cluster.py:3648, query) 2026-10-29 19:54:54.930000 [ 684 ] DEBUG : Executing query SELECT COUNT(*) from someothertable on node4 (cluster.py:3648, query) 2026-10-29 19:54:55.943000 [ 684 ] DEBUG : Executing query TRUNCATE TABLE someothertable on node3 (cluster.py:3648, query) ________________ test_distributed_insert_select_with_replicated ________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_distributed_insert_select_with_replicated(started_cluster): first_replica_first_shard = started_cluster.instances["s0_0_0"] second_replica_first_shard = started_cluster.instances["s0_0_1"] > first_replica_first_shard.query( """DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC;""" ) test_s3_cluster/test.py:290: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3649: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 231, stderr: Received exception from server (version 25.3.14): E Code: 999. DB::Exception: Received from 172.16.5.11:9000. Coordination::Exception. Coordination::Exception: Coordination error: Connection loss, path /clickhouse/task_queue/ddl/query-. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb E 3. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type, std::type_identity::type>, char const*&&, String const&) @ 0x0000000010ef6204 E 4. Coordination::Exception::fromPath(Coordination::Error, String const&) @ 0x0000000010ef541f E 5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:470: zkutil::ZooKeeper::createAncestors(String const&) @ 0x00000000315ef6ec E 6. ./build_docker/./src/Interpreters/DDLWorker.cpp:1087: DB::DDLWorker::enqueueQueryAttempt(DB::DDLLogEntry&) @ 0x0000000029ad46ac E 7. ./build_docker/./src/Interpreters/DDLWorker.cpp:1073: DB::DDLWorker::enqueueQuery(DB::DDLLogEntry&, DB::ZooKeeperRetriesInfo const&) @ 0x0000000029ad3b7c E 8. ./build_docker/./src/Interpreters/executeDDLQueryOnCluster.cpp:192: DB::executeDDLQueryOnCluster(std::shared_ptr const&, std::shared_ptr, DB::DDLQueryOnClusterParams const&) @ 0x000000002af10b87 E 9. ./build_docker/./src/Interpreters/InterpreterDropQuery.cpp:84: DB::InterpreterDropQuery::executeSingleDropQuery(std::shared_ptr const&) @ 0x000000002a6ba151 E 10. ./build_docker/./src/Interpreters/InterpreterDropQuery.cpp:72: DB::InterpreterDropQuery::execute() @ 0x000000002a6b97cd E 11. ./build_docker/./src/Interpreters/executeQuery.cpp:1457: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af36de3 E 12. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 E 13. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 E 14. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa E 15. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f E 16. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 E 17. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b E 18. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 E 19. asan_thread_start(void*) @ 0x000000000c676e37 E 20. ? @ 0x00007f4c97714ac3 E 21. ? @ 0x00007f4c977a6850 E . (KEEPER_EXCEPTION) E (query: DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC;) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC; on s0_0_0 Executing query DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC; on s0_0_0 ------------------------------ Captured log call ------------------------------- 2026-10-29 19:55:09.926000 [ 681 ] DEBUG : Executing query DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC; on s0_0_0 (cluster.py:3648, query) ________ test_parallel_distributed_insert_select_with_schema_inference _________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_parallel_distributed_insert_select_with_schema_inference(started_cluster): node = started_cluster.instances["s0_0_0"] > node.query( """DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC;""" ) test_s3_cluster/test.py:357: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3649: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 231, stderr: Received exception from server (version 25.3.14): E Code: 999. DB::Exception: Received from 172.16.5.11:9000. Coordination::Exception. Coordination::Exception: Coordination error: Operation timeout, path /clickhouse/task_queue/ddl/query-. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb E 3. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type, std::type_identity::type>, char const*&&, String const&) @ 0x0000000010ef6204 E 4. Coordination::Exception::fromPath(Coordination::Error, String const&) @ 0x0000000010ef541f E 5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:470: zkutil::ZooKeeper::createAncestors(String const&) @ 0x00000000315ef6ec E 6. ./build_docker/./src/Interpreters/DDLWorker.cpp:1087: DB::DDLWorker::enqueueQueryAttempt(DB::DDLLogEntry&) @ 0x0000000029ad46ac E 7. ./build_docker/./src/Interpreters/DDLWorker.cpp:1073: DB::DDLWorker::enqueueQuery(DB::DDLLogEntry&, DB::ZooKeeperRetriesInfo const&) @ 0x0000000029ad3b7c E 8. ./build_docker/./src/Interpreters/executeDDLQueryOnCluster.cpp:192: DB::executeDDLQueryOnCluster(std::shared_ptr const&, std::shared_ptr, DB::DDLQueryOnClusterParams const&) @ 0x000000002af10b87 E 9. ./build_docker/./src/Interpreters/InterpreterDropQuery.cpp:84: DB::InterpreterDropQuery::executeSingleDropQuery(std::shared_ptr const&) @ 0x000000002a6ba151 E 10. ./build_docker/./src/Interpreters/InterpreterDropQuery.cpp:72: DB::InterpreterDropQuery::execute() @ 0x000000002a6b97cd E 11. ./build_docker/./src/Interpreters/executeQuery.cpp:1457: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af36de3 E 12. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 E 13. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 E 14. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa E 15. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f E 16. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 E 17. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b E 18. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 E 19. asan_thread_start(void*) @ 0x000000000c676e37 E 20. ? @ 0x00007f4c97714ac3 E 21. ? @ 0x00007f4c977a6850 E . (KEEPER_EXCEPTION) E (query: DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC;) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC; on s0_0_0 Executing query DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC; on s0_0_0 ------------------------------ Captured log call ------------------------------- 2026-10-29 19:55:10.948000 [ 681 ] DEBUG : Executing query DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC; on s0_0_0 (cluster.py:3648, query) ___________________________ test_backup_inner_table ____________________________ [gw4] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = cleanup = None def test_backup_inner_table(started_cluster, cleanup): > do_test_backup(False) test_refreshable_mv/test.py:447: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_refreshable_mv/test.py:437: in do_test_backup assert_eq_with_retry( helpers/test_tools.py:103: in assert_eq_with_retry instance.query( helpers/cluster.py:3649: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): E Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb E 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 E 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 E 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 E 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc E 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef E 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 E 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 E 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 E 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f E 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 E 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 E 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 E 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 E 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa E 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f E 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 E 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b E 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 E 21. asan_thread_start(void*) @ 0x000000000c676e37 E 22. ? @ 0x00007fd702210ac3 E 23. ? @ 0x00007fd7022a2850 E . (ABORTED) E (query: select * from re.rmv order by x) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node1 Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node1 Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node2 Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node2 Executing query create table re.src (x Int64) engine ReplicatedMergeTree order by x;insert into re.src values (1); on node1 Executing query create table re.src (x Int64) engine ReplicatedMergeTree order by x;insert into re.src values (1); on node1 Executing query system sync database replica re;system sync replica re.src;create materialized view re.rmv refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select x from re.src;system wait view re.rmv on node2 Executing query system sync database replica re;system sync replica re.src;create materialized view re.rmv refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select x from re.src;system wait view re.rmv on node2 Executing query select * from re.rmv on node2 Executing query select * from re.rmv on node2 Executing query BACKUP ALL EXCEPT DATABASE system ON CLUSTER 'default' TO S3('http://minio1:9001/root/data/backups/backup1', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 Executing query BACKUP ALL EXCEPT DATABASE system ON CLUSTER 'default' TO S3('http://minio1:9001/root/data/backups/backup1', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node1 Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node1 Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node2 Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node2 Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node1 Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node1 Executing query RESTORE ALL ON CLUSTER 'default' FROM S3('http://minio1:9001/root/data/backups/backup1', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 Executing query RESTORE ALL ON CLUSTER 'default' FROM S3('http://minio1:9001/root/data/backups/backup1', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node1 Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node1 Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node2 Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node2 Executing query SYSTEM SYNC REPLICA re.rmv on node1 Executing query SYSTEM SYNC REPLICA re.rmv on node1 Executing query SYSTEM SYNC REPLICA re.rmv on node2 Executing query SYSTEM SYNC REPLICA re.rmv on node2 Executing query SELECT * FROM re.rmv on node1 Executing query SELECT * FROM re.rmv on node1 Executing query SELECT * FROM re.rmv on node2 Executing query SELECT * FROM re.rmv on node2 Executing query insert into re.src values (2) on node1 Executing query insert into re.src values (2) on node1 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 assert_eq_with_retry retry 19 exception Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/test_tools.py", line 83, in assert_eq_with_retry instance.query( File "/ClickHouse/tests/integration/helpers/cluster.py", line 3649, in query return self.client.query( File "/ClickHouse/tests/integration/helpers/client.py", line 39, in wrap return func(self, *args, **kwargs) File "/ClickHouse/tests/integration/helpers/client.py", line 79, in query ).get_answer() File "/ClickHouse/tests/integration/helpers/client.py", line 248, in get_answer raise QueryRuntimeException( helpers.client.QueryRuntimeException: Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) assert_eq_with_retry retry 19 exception Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/test_tools.py", line 83, in assert_eq_with_retry instance.query( File "/ClickHouse/tests/integration/helpers/cluster.py", line 3649, in query return self.client.query( File "/ClickHouse/tests/integration/helpers/client.py", line 39, in wrap return func(self, *args, **kwargs) File "/ClickHouse/tests/integration/helpers/client.py", line 79, in query ).get_answer() File "/ClickHouse/tests/integration/helpers/client.py", line 248, in get_answer raise QueryRuntimeException( helpers.client.QueryRuntimeException: Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 assert_eq_with_retry retry 20 exception Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/test_tools.py", line 83, in assert_eq_with_retry instance.query( File "/ClickHouse/tests/integration/helpers/cluster.py", line 3649, in query return self.client.query( File "/ClickHouse/tests/integration/helpers/client.py", line 39, in wrap return func(self, *args, **kwargs) File "/ClickHouse/tests/integration/helpers/client.py", line 79, in query ).get_answer() File "/ClickHouse/tests/integration/helpers/client.py", line 248, in get_answer raise QueryRuntimeException( helpers.client.QueryRuntimeException: Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) assert_eq_with_retry retry 20 exception Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/test_tools.py", line 83, in assert_eq_with_retry instance.query( File "/ClickHouse/tests/integration/helpers/cluster.py", line 3649, in query return self.client.query( File "/ClickHouse/tests/integration/helpers/client.py", line 39, in wrap return func(self, *args, **kwargs) File "/ClickHouse/tests/integration/helpers/client.py", line 79, in query ).get_answer() File "/ClickHouse/tests/integration/helpers/client.py", line 248, in get_answer raise QueryRuntimeException( helpers.client.QueryRuntimeException: Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) Executing query select * from re.rmv order by x on node2 Executing query select * from re.rmv order by x on node2 ------------------------------ Captured log call ------------------------------- 2026-10-29 19:54:35.031000 [ 687 ] DEBUG : Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node1 (cluster.py:3648, query) 2026-10-29 19:54:35.809000 [ 687 ] DEBUG : Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node2 (cluster.py:3648, query) 2026-10-29 19:54:36.388000 [ 687 ] DEBUG : Executing query create table re.src (x Int64) engine ReplicatedMergeTree order by x;insert into re.src values (1); on node1 (cluster.py:3648, query) 2026-10-29 19:54:37.436000 [ 687 ] DEBUG : Executing query system sync database replica re;system sync replica re.src;create materialized view re.rmv refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select x from re.src;system wait view re.rmv on node2 (cluster.py:3648, query) 2026-10-29 19:54:38.677000 [ 687 ] DEBUG : Executing query select * from re.rmv on node2 (cluster.py:3648, query) 2026-10-29 19:54:39.150000 [ 687 ] DEBUG : Executing query BACKUP ALL EXCEPT DATABASE system ON CLUSTER 'default' TO S3('http://minio1:9001/root/data/backups/backup1', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 (cluster.py:3648, query) 2026-10-29 19:54:46.179000 [ 687 ] DEBUG : Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node1 (cluster.py:3648, query) 2026-10-29 19:54:48.939000 [ 687 ] DEBUG : Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node2 (cluster.py:3648, query) 2026-10-29 19:54:49.829000 [ 687 ] DEBUG : Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node1 (cluster.py:3648, query) 2026-10-29 19:54:53.919000 [ 687 ] DEBUG : Executing query RESTORE ALL ON CLUSTER 'default' FROM S3('http://minio1:9001/root/data/backups/backup1', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 (cluster.py:3648, query) 2026-10-29 19:55:07.680000 [ 687 ] DEBUG : Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node1 (cluster.py:3648, query) 2026-10-29 19:55:08.515000 [ 687 ] DEBUG : Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node2 (cluster.py:3648, query) 2026-10-29 19:55:09.416000 [ 687 ] DEBUG : Executing query SYSTEM SYNC REPLICA re.rmv on node1 (cluster.py:3648, query) 2026-10-29 19:55:10.017000 [ 687 ] DEBUG : Executing query SYSTEM SYNC REPLICA re.rmv on node2 (cluster.py:3648, query) 2026-10-29 19:55:10.531000 [ 687 ] DEBUG : Executing query SELECT * FROM re.rmv on node1 (cluster.py:3648, query) 2026-10-29 19:55:10.900000 [ 687 ] DEBUG : Executing query SELECT * FROM re.rmv on node2 (cluster.py:3648, query) 2026-10-29 19:55:23.586000 [ 687 ] DEBUG : Executing query insert into re.src values (2) on node1 (cluster.py:3648, query) 2026-10-29 19:55:25.063000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:26.998000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:31.606000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:32.819000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:33.965000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:34.913000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:36.002000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:36.894000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:38.090000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:39.060000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:40.218000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:41.116000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:42.089000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:45.154000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:47.258000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:48.367000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:49.967000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:51.227000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:52.461000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:53.897000 [ 687 ] ERROR : assert_eq_with_retry retry 19 exception Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) (test_tools.py:98, assert_eq_with_retry) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/test_tools.py", line 83, in assert_eq_with_retry instance.query( File "/ClickHouse/tests/integration/helpers/cluster.py", line 3649, in query return self.client.query( File "/ClickHouse/tests/integration/helpers/client.py", line 39, in wrap return func(self, *args, **kwargs) File "/ClickHouse/tests/integration/helpers/client.py", line 79, in query ).get_answer() File "/ClickHouse/tests/integration/helpers/client.py", line 248, in get_answer raise QueryRuntimeException( helpers.client.QueryRuntimeException: Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) 2026-10-29 19:55:54.407000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) 2026-10-29 19:55:55.818000 [ 687 ] ERROR : assert_eq_with_retry retry 20 exception Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) (test_tools.py:98, assert_eq_with_retry) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/test_tools.py", line 83, in assert_eq_with_retry instance.query( File "/ClickHouse/tests/integration/helpers/cluster.py", line 3649, in query return self.client.query( File "/ClickHouse/tests/integration/helpers/client.py", line 39, in wrap return func(self, *args, **kwargs) File "/ClickHouse/tests/integration/helpers/client.py", line 79, in query ).get_answer() File "/ClickHouse/tests/integration/helpers/client.py", line 248, in get_answer raise QueryRuntimeException( helpers.client.QueryRuntimeException: Client failed! Return code: 236, stderr: Received exception from server (version 25.3.14): Code: 236. DB::Exception: Received from 172.16.6.10:9000. DB::Exception: Shutdown is called for table. Stack trace: 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000c6de7b2 4. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:9241: DB::StorageReplicatedMergeTree::waitForProcessingQueue(unsigned long, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000002e9c1961 5. ./build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:1241: DB::InterpreterSystemQuery::trySyncReplica(std::shared_ptr, DB::SyncReplicaMode, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr) @ 0x000000002afbeff7 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:1002: DB::RefreshTask::getAndLockTargetTable(DB::StorageID const&, std::shared_ptr const&) @ 0x000000002fd56fcc 7. ./build_docker/./src/Storages/StorageMaterializedView.cpp:344: DB::StorageMaterializedView::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000002e70b9ef 8. ./build_docker/./src/Planner/PlannerJoinTree.cpp:950: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000002a7f5048 9. ./build_docker/./src/Planner/PlannerJoinTree.cpp:2257: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000002a7e57c2 10. ./build_docker/./src/Planner/Planner.cpp:1560: DB::Planner::buildPlanForQueryNode() @ 0x000000002a7b4217 11. ./build_docker/./src/Planner/Planner.cpp:1315: DB::Planner::buildQueryPlanIfNeeded() @ 0x000000002a7ac53f 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:255: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x000000002a7a7712 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1405: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af35e99 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 15. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 16. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 19. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b 20. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 21. asan_thread_start(void*) @ 0x000000000c676e37 22. ? @ 0x00007fd702210ac3 23. ? @ 0x00007fd7022a2850 . (ABORTED) (query: select * from re.rmv order by x) 2026-10-29 19:55:56.319000 [ 687 ] DEBUG : Executing query select * from re.rmv order by x on node2 (cluster.py:3648, query) --------------------------- Captured stderr teardown --------------------------- Executing query drop database if exists re sync;drop table if exists system.a; on node1 Executing query drop database if exists re sync;drop table if exists system.a; on node1 Executing query drop database if exists re sync;drop table if exists system.a; on node2 Executing query drop database if exists re sync;drop table if exists system.a; on node2 Executing query drop database if exists re sync;drop table if exists system.a; on reading_node Executing query drop database if exists re sync;drop table if exists system.a; on reading_node ---------------------------- Captured log teardown ----------------------------- 2026-10-29 19:55:57.248000 [ 687 ] DEBUG : Executing query drop database if exists re sync;drop table if exists system.a; on node1 (cluster.py:3648, query) 2026-10-29 19:56:45.512000 [ 687 ] DEBUG : Executing query drop database if exists re sync;drop table if exists system.a; on node2 (cluster.py:3648, query) 2026-10-29 19:57:25.780000 [ 687 ] DEBUG : Executing query drop database if exists re sync;drop table if exists system.a; on reading_node (cluster.py:3648, query) _ test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] _ [gw5] linux -- Python 3.10.12 /usr/bin/python3 start_cluster = cluster = 'test_single_shard_multiple_replicas', max_parallel_replicas = 4 prefer_localhost_replica = 1 @pytest.mark.parametrize( "cluster,max_parallel_replicas,prefer_localhost_replica", [ pytest.param("test_single_shard_multiple_replicas", 2, 0), pytest.param("test_single_shard_multiple_replicas", 3, 0), pytest.param("test_single_shard_multiple_replicas", 4, 0), pytest.param("test_single_shard_multiple_replicas", 10, 0), pytest.param("test_single_shard_multiple_replicas", 2, 1), pytest.param("test_single_shard_multiple_replicas", 3, 1), pytest.param("test_single_shard_multiple_replicas", 4, 1), pytest.param("test_single_shard_multiple_replicas", 10, 1), pytest.param("test_multiple_shards_multiple_replicas", 2, 0), pytest.param("test_multiple_shards_multiple_replicas", 3, 0), pytest.param("test_multiple_shards_multiple_replicas", 4, 0), pytest.param("test_multiple_shards_multiple_replicas", 10, 0), pytest.param("test_multiple_shards_multiple_replicas", 2, 1), pytest.param("test_multiple_shards_multiple_replicas", 3, 1), pytest.param("test_multiple_shards_multiple_replicas", 4, 1), pytest.param("test_multiple_shards_multiple_replicas", 10, 1), ], ) def test_parallel_replicas_over_distributed( start_cluster, cluster, max_parallel_replicas, prefer_localhost_replica ): table_name = "test_table" > create_tables(cluster, table_name) test_parallel_replicas_over_distributed/test.py:128: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_parallel_replicas_over_distributed/test.py:39: in create_tables nodes[3].query( helpers/cluster.py:3649: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 231, stderr: Received exception from server (version 25.3.14): E Code: 999. DB::Exception: Received from 172.16.7.9:9000. Coordination::Exception. Coordination::Exception: Session expired. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 E 2. ./src/Common/Exception.h:112: Coordination::Exception::Exception(T&&, Coordination::Error) @ 0x0000000031663f2d E 3. ./src/Common/ZooKeeper/KeeperException.h:58: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000316f3613 E 4. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:1428: Coordination::ZooKeeper::create(String const&, String const&, bool, bool, std::vector> const&, std::function) @ 0x00000000316f776b E 5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:1204: zkutil::ZooKeeper::asyncTryCreateNoThrow(String const&, String const&, int) @ 0x00000000315ed83b E 6. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:385: zkutil::ZooKeeper::createImpl(String const&, String const&, int, String&) @ 0x00000000315eccc3 E 7. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:484: zkutil::ZooKeeper::createAncestors(String const&) @ 0x00000000315ef2a3 E 8. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:942: DB::StorageReplicatedMergeTree::createTableIfNotExistsAttempt(std::shared_ptr const&, std::shared_ptr) const @ 0x000000002e82147a E 9. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:930: DB::StorageReplicatedMergeTree::createTableIfNotExists(std::shared_ptr const&, DB::ZooKeeperRetriesInfo const&) const @ 0x000000002e8094d4 E 10. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:594: DB::StorageReplicatedMergeTree::StorageReplicatedMergeTree(DB::TableZnodeInfo const&, DB::LoadingStrictnessLevel, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, std::shared_ptr, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::ZooKeeperRetriesInfo const&) @ 0x000000002e803c66 E 11. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:41: std::shared_ptr std::allocate_shared[abi:ne190107], DB::TableZnodeInfo&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr>, bool&, DB::ZooKeeperRetriesInfo&, 0>(std::allocator const&, DB::TableZnodeInfo&, DB::LoadingStrictnessLevel const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, std::shared_ptr&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr>&&, bool&, DB::ZooKeeperRetriesInfo&) @ 0x000000002fa15fe1 E 12. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:851: DB::create(DB::StorageFactory::Arguments const&) @ 0x000000002fa0ed02 E 13. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000002e568df2 E 14. ./build_docker/./src/Interpreters/InterpreterCreateQuery.cpp:1945: DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr>&, DB::LoadingStrictnessLevel) @ 0x000000002a68a548 E 15. ./build_docker/./src/Interpreters/InterpreterCreateQuery.cpp:1743: DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x000000002a67bbcd E 16. ./build_docker/./src/Interpreters/InterpreterCreateQuery.cpp:2355: DB::InterpreterCreateQuery::execute() @ 0x000000002a696498 E 17. ./build_docker/./src/Interpreters/executeQuery.cpp:1457: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af36de3 E 18. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 E 19. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 E 20. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa E 21. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f E 22. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 E 23. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b E 24. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 E 25. asan_thread_start(void*) @ 0x000000000c676e37 E 26. ? @ 0x00007fc9448d4ac3 E 27. ? @ 0x00007fc944966850 E . (KEEPER_EXCEPTION) E (query: CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r4') ORDER BY (key)) helpers/client.py:248: QueryRuntimeException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- Running tests in /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/test.py Running tests in /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/test.py Cluster start called. is_up=False Cluster start called. is_up=False Docker networks for project roottestparallelreplicasoverdistributed-gw5 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestparallelreplicasoverdistributed-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestparallelreplicasoverdistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestparallelreplicasoverdistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestparallelreplicasoverdistributed-gw5 are DRIVER VOLUME NAME Docker volumes for project roottestparallelreplicasoverdistributed-gw5 are DRIVER VOLUME NAME Cleanup called Cleanup called Docker networks for project roottestparallelreplicasoverdistributed-gw5 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestparallelreplicasoverdistributed-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestparallelreplicasoverdistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestparallelreplicasoverdistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestparallelreplicasoverdistributed-gw5 are DRIVER VOLUME NAME Docker volumes for project roottestparallelreplicasoverdistributed-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestparallelreplicasoverdistributed-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestparallelreplicasoverdistributed-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestparallelreplicasoverdistributed-gw5 No running containers for project: roottestparallelreplicasoverdistributed-gw5 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:10 Stdout:10 Command:[docker volume prune -f] Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Volumes pruned: 10 Volumes pruned: 10 Setup directory for instance: n1 Setup directory for instance: n1 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/configs/config.d Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/database Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/database Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/logs Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: n2 Setup directory for instance: n2 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/configs/config.d Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/database Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/database Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/logs Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: n3 Setup directory for instance: n3 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/configs/config.d Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/database Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/database Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/logs Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: n4 Setup directory for instance: n4 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/configs/config.d Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/configs/config.d Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/database Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/database Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/logs Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: n5 Setup directory for instance: n5 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/configs/config.d Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/configs/config.d Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/database Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/database Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/logs Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: n6 Setup directory for instance: n6 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/configs/config.d Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/configs/config.d Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/database Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/database Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/logs Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml pull] Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml pull] Stderr: n1 Skipped - Image is already being pulled by n5 Stderr: n1 Skipped - Image is already being pulled by n5 Stderr: n2 Skipped - Image is already being pulled by n5 Stderr: n2 Skipped - Image is already being pulled by n5 Stderr: zoo2 Skipped - Image is already being pulled by n5 Stderr: zoo2 Skipped - Image is already being pulled by n5 Stderr: zoo1 Skipped - Image is already being pulled by n5 Stderr: zoo1 Skipped - Image is already being pulled by n5 Stderr: n4 Skipped - Image is already being pulled by n5 Stderr: n4 Skipped - Image is already being pulled by n5 Stderr: n6 Skipped - Image is already being pulled by n5 Stderr: n6 Skipped - Image is already being pulled by n5 Stderr: zoo3 Skipped - Image is already being pulled by n5 Stderr: zoo3 Skipped - Image is already being pulled by n5 Stderr: n3 Skipped - Image is already being pulled by n5 Stderr: n3 Skipped - Image is already being pulled by n5 Stderr: n5 Pulling Stderr: n5 Pulling Stderr: n5 Pulled Stderr: n5 Pulled Setup ZooKeeper Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/log', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/config', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/coordination', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/log', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/config', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/coordination', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/log', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/config', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/coordination'] Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/log', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/config', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/coordination', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/log', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/config', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/coordination', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/log', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/config', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/coordination'] Command:[docker compose --project-name roottestparallelreplicasoverdistributed-gw5 --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Command:[docker compose --project-name roottestparallelreplicasoverdistributed-gw5 --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr:time="2026-10-29T19:55:10Z" level=trace msg="Docker Desktop integration not enabled" Stderr:time="2026-10-29T19:55:10Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Creating Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Creating Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Created Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Creating Stderr:time="2026-10-29T19:55:15Z" level=debug msg="otel error" error="" Stderr:time="2026-10-29T19:55:15Z" level=debug msg="otel error" error="" Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Started Stderr:time="2026-10-29T19:55:25Z" level=debug msg="otel error" error="" Stderr:time="2026-10-29T19:55:25Z" level=debug msg="otel error" error="" Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Started Stderr:time="2026-10-29T19:55:26Z" level=debug msg="otel error" error="" Stderr:time="2026-10-29T19:55:26Z" level=debug msg="otel error" error="" Stderr:time="2026-10-29T19:55:26Z" level=debug msg="otel error" error="" Stderr:time="2026-10-29T19:55:26Z" level=debug msg="otel error" error="" Wait ZooKeeper to start Wait ZooKeeper to start get_instance_ip instance_name=zoo1 get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.7.4, port:2181, use_ssl:False get_kazoo_client: zoo1, ip:172.16.7.4, port:2181, use_ssl:False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED Can't connect to ZooKeeper zoo1: Connection time-out Can't connect to ZooKeeper zoo1: Connection time-out get_instance_ip instance_name=zoo1 get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.7.4, port:2181, use_ssl:False get_kazoo_client: zoo1, ip:172.16.7.4, port:2181, use_ssl:False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.7.2, port:2181, use_ssl:False get_kazoo_client: zoo2, ip:172.16.7.2, port:2181, use_ssl:False Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.7.3, port:2181, use_ssl:False get_kazoo_client: zoo3, ip:172.16.7.3, port:2181, use_ssl:False Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml up -d --no-recreate') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml up -d --no-recreate] Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml up -d --no-recreate] Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Running Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Running Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Running Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Running Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Running Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Running Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Creating Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Created Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Starting Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Started Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Started ClickHouse instance created ClickHouse instance created get_instance_ip instance_name=n1 get_instance_ip instance_name=n1 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=n1 get_instance_ip instance_name=n1 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in n1, ip: 172.16.7.6... Waiting for ClickHouse start in n1, ip: 172.16.7.6... http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None ClickHouse n1 started ClickHouse n1 started get_instance_ip instance_name=n2 get_instance_ip instance_name=n2 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=n2 get_instance_ip instance_name=n2 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in n2, ip: 172.16.7.8... Waiting for ClickHouse start in n2, ip: 172.16.7.8... http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None ClickHouse n2 started ClickHouse n2 started get_instance_ip instance_name=n3 get_instance_ip instance_name=n3 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=n3 get_instance_ip instance_name=n3 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in n3, ip: 172.16.7.10... Waiting for ClickHouse start in n3, ip: 172.16.7.10... http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/934a2003e4be25212969742f57aeb9b0f3ce9cf22fcd27c54654dcf7ec47e740/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/934a2003e4be25212969742f57aeb9b0f3ce9cf22fcd27c54654dcf7ec47e740/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/934a2003e4be25212969742f57aeb9b0f3ce9cf22fcd27c54654dcf7ec47e740/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/934a2003e4be25212969742f57aeb9b0f3ce9cf22fcd27c54654dcf7ec47e740/json HTTP/1.1" 200 None ClickHouse n3 started ClickHouse n3 started get_instance_ip instance_name=n4 get_instance_ip instance_name=n4 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n4-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n4-1/json HTTP/1.1" 200 None get_instance_ip instance_name=n4 get_instance_ip instance_name=n4 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n4-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n4-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in n4, ip: 172.16.7.9... Waiting for ClickHouse start in n4, ip: 172.16.7.9... http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n4-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n4-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/7c956c8d1d31acc3ccb25b99b9c7bde8c8ae87c75ed23ece00210b734e09b9db/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/7c956c8d1d31acc3ccb25b99b9c7bde8c8ae87c75ed23ece00210b734e09b9db/json HTTP/1.1" 200 None ClickHouse n4 started ClickHouse n4 started get_instance_ip instance_name=n5 get_instance_ip instance_name=n5 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n5-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n5-1/json HTTP/1.1" 200 None get_instance_ip instance_name=n5 get_instance_ip instance_name=n5 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n5-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n5-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in n5, ip: 172.16.7.7... Waiting for ClickHouse start in n5, ip: 172.16.7.7... http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n5-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n5-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/71a86bec69c89f34c049406dc0c5b8ce9fb1adeababb7ccbbfcad4a59be08d0a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/71a86bec69c89f34c049406dc0c5b8ce9fb1adeababb7ccbbfcad4a59be08d0a/json HTTP/1.1" 200 None ClickHouse n5 started ClickHouse n5 started get_instance_ip instance_name=n6 get_instance_ip instance_name=n6 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n6-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n6-1/json HTTP/1.1" 200 None get_instance_ip instance_name=n6 get_instance_ip instance_name=n6 http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n6-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n6-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in n6, ip: 172.16.7.5... Waiting for ClickHouse start in n6, ip: 172.16.7.5... http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n6-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n6-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c297f0eb6a800266b26fef0cc13fe8210145b85020112c5d4603f09c4a1adbc7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c297f0eb6a800266b26fef0cc13fe8210145b85020112c5d4603f09c4a1adbc7/json HTTP/1.1" 200 None ClickHouse n6 started ClickHouse n6 started ------------------------------ Captured log setup ------------------------------ 2026-10-29 19:55:07.441000 [ 690 ] INFO : Running tests in /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/test.py (cluster.py:2738, start) 2026-10-29 19:55:07.441000 [ 690 ] DEBUG : Cluster start called. is_up=False (cluster.py:2745, start) 2026-10-29 19:55:07.558000 [ 690 ] DEBUG : Docker networks for project roottestparallelreplicasoverdistributed-gw5 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 19:55:07.670000 [ 690 ] DEBUG : Docker containers for project roottestparallelreplicasoverdistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 19:55:07.712000 [ 690 ] DEBUG : Docker volumes for project roottestparallelreplicasoverdistributed-gw5 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 19:55:07.713000 [ 690 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 19:55:07.794000 [ 690 ] DEBUG : Docker networks for project roottestparallelreplicasoverdistributed-gw5 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 19:55:07.914000 [ 690 ] DEBUG : Docker containers for project roottestparallelreplicasoverdistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 19:55:07.960000 [ 690 ] DEBUG : Docker volumes for project roottestparallelreplicasoverdistributed-gw5 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 19:55:07.960000 [ 690 ] DEBUG : Command:[docker container list --all --filter name='^/roottestparallelreplicasoverdistributed-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 19:55:08.031000 [ 690 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 19:55:08.031000 [ 690 ] DEBUG : No running containers for project: roottestparallelreplicasoverdistributed-gw5 (cluster.py:879, cleanup) 2026-10-29 19:55:08.031000 [ 690 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 19:55:08.053000 [ 690 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 19:55:08.053000 [ 690 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:55:08.080000 [ 690 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:55:08.081000 [ 690 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 19:55:08.081000 [ 690 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 19:55:08.081000 [ 690 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 19:55:08.153000 [ 690 ] DEBUG : Stdout:10 (cluster.py:145, run_and_check) 2026-10-29 19:55:08.153000 [ 690 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:55:08.176000 [ 690 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:55:08.176000 [ 690 ] DEBUG : Volumes pruned: 10 (cluster.py:915, cleanup) 2026-10-29 19:55:08.177000 [ 690 ] DEBUG : Setup directory for instance: n1 (cluster.py:2758, start) 2026-10-29 19:55:08.178000 [ 690 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:55:08.178000 [ 690 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:55:08.178000 [ 690 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:55:08.179000 [ 690 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:55:08.180000 [ 690 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:55:08.181000 [ 690 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/database (cluster.py:4758, create_dir) 2026-10-29 19:55:08.181000 [ 690 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/logs (cluster.py:4769, create_dir) 2026-10-29 19:55:08.181000 [ 690 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:55:08.181000 [ 690 ] DEBUG : Setup directory for instance: n2 (cluster.py:2758, start) 2026-10-29 19:55:08.182000 [ 690 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:55:08.182000 [ 690 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:55:08.182000 [ 690 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:55:08.182000 [ 690 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:55:08.183000 [ 690 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:55:08.183000 [ 690 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/database (cluster.py:4758, create_dir) 2026-10-29 19:55:08.183000 [ 690 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/logs (cluster.py:4769, create_dir) 2026-10-29 19:55:08.183000 [ 690 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:55:08.183000 [ 690 ] DEBUG : Setup directory for instance: n3 (cluster.py:2758, start) 2026-10-29 19:55:08.184000 [ 690 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:55:08.184000 [ 690 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:55:08.184000 [ 690 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:55:08.184000 [ 690 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:55:08.184000 [ 690 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:55:08.185000 [ 690 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/database (cluster.py:4758, create_dir) 2026-10-29 19:55:08.185000 [ 690 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/logs (cluster.py:4769, create_dir) 2026-10-29 19:55:08.185000 [ 690 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:55:08.185000 [ 690 ] DEBUG : Setup directory for instance: n4 (cluster.py:2758, start) 2026-10-29 19:55:08.185000 [ 690 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:55:08.185000 [ 690 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:55:08.185000 [ 690 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:55:08.186000 [ 690 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:55:08.186000 [ 690 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:55:08.186000 [ 690 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/database (cluster.py:4758, create_dir) 2026-10-29 19:55:08.186000 [ 690 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/logs (cluster.py:4769, create_dir) 2026-10-29 19:55:08.186000 [ 690 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:55:08.186000 [ 690 ] DEBUG : Setup directory for instance: n5 (cluster.py:2758, start) 2026-10-29 19:55:08.187000 [ 690 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:55:08.187000 [ 690 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:55:08.187000 [ 690 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:55:08.187000 [ 690 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:55:08.188000 [ 690 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:55:08.188000 [ 690 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/database (cluster.py:4758, create_dir) 2026-10-29 19:55:08.188000 [ 690 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/logs (cluster.py:4769, create_dir) 2026-10-29 19:55:08.188000 [ 690 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:55:08.188000 [ 690 ] DEBUG : Setup directory for instance: n6 (cluster.py:2758, start) 2026-10-29 19:55:08.188000 [ 690 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:55:08.188000 [ 690 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:55:08.189000 [ 690 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:55:08.189000 [ 690 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:55:08.189000 [ 690 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:55:08.189000 [ 690 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/database (cluster.py:4758, create_dir) 2026-10-29 19:55:08.189000 [ 690 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/logs (cluster.py:4769, create_dir) 2026-10-29 19:55:08.190000 [ 690 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:55:08.190000 [ 690 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env (cluster.py:96, _create_env_file) 2026-10-29 19:55:08.194000 [ 690 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 19:55:08.194000 [ 690 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 19:55:08.194000 [ 690 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 19:55:08.194000 [ 690 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 19:55:08.260000 [ 690 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2026-10-29 19:55:08.262000 [ 690 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Stderr: n1 Skipped - Image is already being pulled by n5 (cluster.py:147, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Stderr: n2 Skipped - Image is already being pulled by n5 (cluster.py:147, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Stderr: zoo2 Skipped - Image is already being pulled by n5 (cluster.py:147, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Stderr: zoo1 Skipped - Image is already being pulled by n5 (cluster.py:147, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Stderr: n4 Skipped - Image is already being pulled by n5 (cluster.py:147, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Stderr: n6 Skipped - Image is already being pulled by n5 (cluster.py:147, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Stderr: zoo3 Skipped - Image is already being pulled by n5 (cluster.py:147, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Stderr: n3 Skipped - Image is already being pulled by n5 (cluster.py:147, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Stderr: n5 Pulling (cluster.py:147, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Stderr: n5 Pulled (cluster.py:147, run_and_check) 2026-10-29 19:55:09.873000 [ 690 ] DEBUG : Setup ZooKeeper (cluster.py:2799, start) 2026-10-29 19:55:09.874000 [ 690 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/log', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/config', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper1/coordination', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/log', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/config', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper2/coordination', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/log', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/config', '/ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/keeper3/coordination'] (cluster.py:2800, start) 2026-10-29 19:55:09.875000 [ 690 ] DEBUG : Command:[docker compose --project-name roottestparallelreplicasoverdistributed-gw5 --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] (cluster.py:121, run_and_check) 2026-10-29 19:55:26.054000 [ 690 ] DEBUG : Stderr:time="2026-10-29T19:55:10Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:147, run_and_check) 2026-10-29 19:55:26.056000 [ 690 ] DEBUG : Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Creating (cluster.py:147, run_and_check) 2026-10-29 19:55:26.056000 [ 690 ] DEBUG : Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Created (cluster.py:147, run_and_check) 2026-10-29 19:55:26.056000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:55:26.056000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:55:26.056000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:55:26.056000 [ 690 ] DEBUG : Stderr:time="2026-10-29T19:55:15Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr:time="2026-10-29T19:55:25Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr:time="2026-10-29T19:55:26Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-10-29 19:55:26.057000 [ 690 ] DEBUG : Stderr:time="2026-10-29T19:55:26Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-10-29 19:55:26.058000 [ 690 ] DEBUG : Wait ZooKeeper to start (cluster.py:2436, wait_zookeeper_to_start) 2026-10-29 19:55:26.059000 [ 690 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2005, get_instance_ip) 2026-10-29 19:55:26.090000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:26.139000 [ 690 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.7.4, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-10-29 19:55:26.201000 [ 690 ] INFO : Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:26.202000 [ 690 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 19:55:26.317000 [ 690 ] INFO : Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:26.317000 [ 690 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 19:55:26.448000 [ 690 ] INFO : Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:26.448000 [ 690 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 19:55:26.619000 [ 690 ] INFO : Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:26.620000 [ 690 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 19:55:27.015000 [ 690 ] INFO : Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:27.015000 [ 690 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 19:55:27.825000 [ 690 ] INFO : Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:30.839000 [ 690 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 19:55:32.286000 [ 690 ] INFO : Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:32.286000 [ 690 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 19:55:35.433000 [ 690 ] INFO : Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:35.434000 [ 690 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 19:55:41.246000 [ 690 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-10-29 19:55:41.247000 [ 690 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-10-29 19:55:41.249000 [ 690 ] DEBUG : Can't connect to ZooKeeper zoo1: Connection time-out (cluster.py:2455, wait_zookeeper_nodes_to_start) 2026-10-29 19:55:41.755000 [ 690 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2005, get_instance_ip) 2026-10-29 19:55:41.787000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:41.790000 [ 690 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.7.4, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-10-29 19:55:41.793000 [ 690 ] INFO : Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:41.795000 [ 690 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-10-29 19:55:41.803000 [ 690 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-10-29 19:55:41.805000 [ 690 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-10-29 19:55:41.808000 [ 690 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-10-29 19:55:41.809000 [ 690 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-10-29 19:55:41.814000 [ 690 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-10-29 19:55:41.815000 [ 690 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-10-29 19:55:41.815000 [ 690 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-10-29 19:55:41.915000 [ 690 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-10-29 19:55:41.915000 [ 690 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-10-29 19:55:41.916000 [ 690 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:2005, get_instance_ip) 2026-10-29 19:55:41.918000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:41.919000 [ 690 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.7.2, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-10-29 19:55:41.920000 [ 690 ] INFO : Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:41.920000 [ 690 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-10-29 19:55:41.927000 [ 690 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-10-29 19:55:41.927000 [ 690 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-10-29 19:55:41.928000 [ 690 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-10-29 19:55:41.928000 [ 690 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-10-29 19:55:41.935000 [ 690 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-10-29 19:55:41.935000 [ 690 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-10-29 19:55:41.935000 [ 690 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-10-29 19:55:42.024000 [ 690 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-10-29 19:55:42.024000 [ 690 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-10-29 19:55:42.025000 [ 690 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:2005, get_instance_ip) 2026-10-29 19:55:42.027000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-zoo3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:42.028000 [ 690 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.7.3, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-10-29 19:55:42.029000 [ 690 ] INFO : Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 19:55:42.029000 [ 690 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-10-29 19:55:42.035000 [ 690 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-10-29 19:55:42.036000 [ 690 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-10-29 19:55:42.036000 [ 690 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-10-29 19:55:42.037000 [ 690 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-10-29 19:55:42.040000 [ 690 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-10-29 19:55:42.040000 [ 690 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-10-29 19:55:42.040000 [ 690 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-10-29 19:55:42.128000 [ 690 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-10-29 19:55:42.128000 [ 690 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-10-29 19:55:42.129000 [ 690 ] DEBUG : All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') (cluster.py:2452, wait_zookeeper_nodes_to_start) 2026-10-29 19:55:42.133000 [ 690 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml up -d --no-recreate') (cluster.py:3139, start) 2026-10-29 19:55:42.133000 [ 690 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml up -d --no-recreate] (cluster.py:121, run_and_check) 2026-10-29 19:55:46.713000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Running (cluster.py:147, run_and_check) 2026-10-29 19:55:46.714000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Running (cluster.py:147, run_and_check) 2026-10-29 19:55:46.714000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Running (cluster.py:147, run_and_check) 2026-10-29 19:55:46.714000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:55:46.714000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:55:46.714000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:55:46.714000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:55:46.714000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:55:46.714000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:55:46.714000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:55:46.714000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : ClickHouse instance created (cluster.py:3147, start) 2026-10-29 19:55:46.715000 [ 690 ] DEBUG : get_instance_ip instance_name=n1 (cluster.py:2005, get_instance_ip) 2026-10-29 19:55:46.717000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:46.718000 [ 690 ] DEBUG : get_instance_ip instance_name=n1 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:55:46.719000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:46.720000 [ 690 ] DEBUG : Waiting for ClickHouse start in n1, ip: 172.16.7.6... (cluster.py:3155, start) 2026-10-29 19:55:46.721000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:46.723000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:46.831000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:46.934000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.038000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.142000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.244000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.352000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.476000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.587000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.692000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.797000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.900000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c52069394caea6b987a959a60dfe97b10b3c88c7b980757e77411083ca8610c1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.901000 [ 690 ] DEBUG : ClickHouse n1 started (cluster.py:3159, start) 2026-10-29 19:55:47.901000 [ 690 ] DEBUG : get_instance_ip instance_name=n2 (cluster.py:2005, get_instance_ip) 2026-10-29 19:55:47.903000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.904000 [ 690 ] DEBUG : get_instance_ip instance_name=n2 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:55:47.905000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.906000 [ 690 ] DEBUG : Waiting for ClickHouse start in n2, ip: 172.16.7.8... (cluster.py:3155, start) 2026-10-29 19:55:47.907000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:47.909000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:48.031000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:48.156000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:48.265000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:48.375000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:48.487000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:48.590000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:48.693000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:48.810000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:48.913000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:49.017000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:49.122000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:49.226000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:49.331000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:49.437000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:49.543000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:49.646000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:49.749000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:49.852000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:49.954000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:50.057000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:50.164000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:50.268000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:50.374000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:50.481000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:50.586000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:50.689000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:50.820000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:50.925000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.031000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.136000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.239000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.347000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.458000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.563000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.666000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.768000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.886000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/5232311b45227ba60bb746d945d4e3dea06bec63255169cfc350d50a19237969/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.886000 [ 690 ] DEBUG : ClickHouse n2 started (cluster.py:3159, start) 2026-10-29 19:55:51.887000 [ 690 ] DEBUG : get_instance_ip instance_name=n3 (cluster.py:2005, get_instance_ip) 2026-10-29 19:55:51.888000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.888000 [ 690 ] DEBUG : get_instance_ip instance_name=n3 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:55:51.889000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.890000 [ 690 ] DEBUG : Waiting for ClickHouse start in n3, ip: 172.16.7.10... (cluster.py:3155, start) 2026-10-29 19:55:51.891000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:51.906000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/934a2003e4be25212969742f57aeb9b0f3ce9cf22fcd27c54654dcf7ec47e740/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.010000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/934a2003e4be25212969742f57aeb9b0f3ce9cf22fcd27c54654dcf7ec47e740/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.011000 [ 690 ] DEBUG : ClickHouse n3 started (cluster.py:3159, start) 2026-10-29 19:55:52.011000 [ 690 ] DEBUG : get_instance_ip instance_name=n4 (cluster.py:2005, get_instance_ip) 2026-10-29 19:55:52.012000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n4-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.012000 [ 690 ] DEBUG : get_instance_ip instance_name=n4 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:55:52.013000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n4-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.013000 [ 690 ] DEBUG : Waiting for ClickHouse start in n4, ip: 172.16.7.9... (cluster.py:3155, start) 2026-10-29 19:55:52.014000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n4-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.016000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/7c956c8d1d31acc3ccb25b99b9c7bde8c8ae87c75ed23ece00210b734e09b9db/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.016000 [ 690 ] DEBUG : ClickHouse n4 started (cluster.py:3159, start) 2026-10-29 19:55:52.016000 [ 690 ] DEBUG : get_instance_ip instance_name=n5 (cluster.py:2005, get_instance_ip) 2026-10-29 19:55:52.018000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n5-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.018000 [ 690 ] DEBUG : get_instance_ip instance_name=n5 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:55:52.019000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n5-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.019000 [ 690 ] DEBUG : Waiting for ClickHouse start in n5, ip: 172.16.7.7... (cluster.py:3155, start) 2026-10-29 19:55:52.020000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n5-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.021000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/71a86bec69c89f34c049406dc0c5b8ce9fb1adeababb7ccbbfcad4a59be08d0a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.022000 [ 690 ] DEBUG : ClickHouse n5 started (cluster.py:3159, start) 2026-10-29 19:55:52.022000 [ 690 ] DEBUG : get_instance_ip instance_name=n6 (cluster.py:2005, get_instance_ip) 2026-10-29 19:55:52.023000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n6-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.023000 [ 690 ] DEBUG : get_instance_ip instance_name=n6 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:55:52.024000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n6-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.024000 [ 690 ] DEBUG : Waiting for ClickHouse start in n6, ip: 172.16.7.5... (cluster.py:3155, start) 2026-10-29 19:55:52.029000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestparallelreplicasoverdistributed-gw5-n6-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.031000 [ 690 ] DEBUG : http://localhost:None "GET /v1.46/containers/c297f0eb6a800266b26fef0cc13fe8210145b85020112c5d4603f09c4a1adbc7/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:55:52.031000 [ 690 ] DEBUG : ClickHouse n6 started (cluster.py:3159, start) ----------------------------- Captured stderr call ----------------------------- Executing query DROP TABLE IF EXISTS test_table SYNC on n1 Executing query DROP TABLE IF EXISTS test_table SYNC on n1 Executing query DROP TABLE IF EXISTS test_table SYNC on n2 Executing query DROP TABLE IF EXISTS test_table SYNC on n2 Executing query DROP TABLE IF EXISTS test_table SYNC on n3 Executing query DROP TABLE IF EXISTS test_table SYNC on n3 Executing query DROP TABLE IF EXISTS test_table SYNC on n4 Executing query DROP TABLE IF EXISTS test_table SYNC on n4 Executing query DROP TABLE IF EXISTS test_table SYNC on n5 Executing query DROP TABLE IF EXISTS test_table SYNC on n5 Executing query DROP TABLE IF EXISTS test_table SYNC on n6 Executing query DROP TABLE IF EXISTS test_table SYNC on n6 Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r1') ORDER BY (key) on n1 Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r1') ORDER BY (key) on n1 Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r2') ORDER BY (key) on n2 Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r2') ORDER BY (key) on n2 Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r3') ORDER BY (key) on n3 Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r3') ORDER BY (key) on n3 Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r4') ORDER BY (key) on n4 Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r4') ORDER BY (key) on n4 ------------------------------ Captured log call ------------------------------- 2026-10-29 19:55:52.071000 [ 690 ] DEBUG : Executing query DROP TABLE IF EXISTS test_table SYNC on n1 (cluster.py:3648, query) 2026-10-29 19:55:52.346000 [ 690 ] DEBUG : Executing query DROP TABLE IF EXISTS test_table SYNC on n2 (cluster.py:3648, query) 2026-10-29 19:55:52.933000 [ 690 ] DEBUG : Executing query DROP TABLE IF EXISTS test_table SYNC on n3 (cluster.py:3648, query) 2026-10-29 19:55:53.359000 [ 690 ] DEBUG : Executing query DROP TABLE IF EXISTS test_table SYNC on n4 (cluster.py:3648, query) 2026-10-29 19:55:53.884000 [ 690 ] DEBUG : Executing query DROP TABLE IF EXISTS test_table SYNC on n5 (cluster.py:3648, query) 2026-10-29 19:55:54.260000 [ 690 ] DEBUG : Executing query DROP TABLE IF EXISTS test_table SYNC on n6 (cluster.py:3648, query) 2026-10-29 19:55:54.631000 [ 690 ] DEBUG : Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r1') ORDER BY (key) on n1 (cluster.py:3648, query) 2026-10-29 19:55:56.367000 [ 690 ] DEBUG : Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r2') ORDER BY (key) on n2 (cluster.py:3648, query) 2026-10-29 19:55:57.275000 [ 690 ] DEBUG : Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r3') ORDER BY (key) on n3 (cluster.py:3648, query) 2026-10-29 19:55:57.990000 [ 690 ] DEBUG : Executing query CREATE TABLE IF NOT EXISTS test_table (key Int64, value String) Engine=ReplicatedMergeTree('/test_parallel_replicas/shard1/test_table', 'r4') ORDER BY (key) on n4 (cluster.py:3648, query) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml stop --timeout 20] Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml stop --timeout 20] Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml down --volumes] Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml down --volumes] Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopping Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopped Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Removing Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Removed Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Removed Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Removing Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Removing Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Removed Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Removed Cleanup called Cleanup called Docker networks for project roottestparallelreplicasoverdistributed-gw5 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestparallelreplicasoverdistributed-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestparallelreplicasoverdistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestparallelreplicasoverdistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestparallelreplicasoverdistributed-gw5 are DRIVER VOLUME NAME Docker volumes for project roottestparallelreplicasoverdistributed-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestparallelreplicasoverdistributed-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestparallelreplicasoverdistributed-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestparallelreplicasoverdistributed-gw5 No running containers for project: roottestparallelreplicasoverdistributed-gw5 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:10 Stdout:10 Command:[docker volume prune -f] Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Volumes pruned: 10 Volumes pruned: 10 ---------------------------- Captured log teardown ----------------------------- 2026-10-29 19:57:08.620000 [ 690 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml stop --timeout 20] (cluster.py:121, run_and_check) 2026-10-29 19:57:32.145000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:32.158000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:32.158000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:32.158000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:32.158000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:32.158000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:32.158000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:32.158000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:32.158000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:32.158000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:32.159000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:32.159000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:32.159000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:32.159000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:32.159000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:32.159000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:32.159000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:32.159000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:32.160000 [ 690 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:57:32.263000 [ 690 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:57:32.310000 [ 690 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:57:32.327000 [ 690 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:57:32.351000 [ 690 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:57:32.375000 [ 690 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:57:32.391000 [ 690 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/.env --project-name roottestparallelreplicasoverdistributed-gw5 --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n3/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n4/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n5/docker-compose.yml --file /ClickHouse/tests/integration/test_parallel_replicas_over_distributed/_instances-0-gw5/n6/docker-compose.yml down --volumes] (cluster.py:121, run_and_check) 2026-10-29 19:57:34.723000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:34.723000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:34.723000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:34.723000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:34.723000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:34.723000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n3-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n2-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n1-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n5-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:57:34.724000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n6-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-n4-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:57:34.725000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo2-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:57:34.726000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo3-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:57:34.726000 [ 690 ] DEBUG : Stderr: Container roottestparallelreplicasoverdistributed-gw5-zoo1-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:57:34.726000 [ 690 ] DEBUG : Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Removing (cluster.py:147, run_and_check) 2026-10-29 19:57:34.726000 [ 690 ] DEBUG : Stderr: Network roottestparallelreplicasoverdistributed-gw5_default Removed (cluster.py:147, run_and_check) 2026-10-29 19:57:34.727000 [ 690 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 19:57:34.745000 [ 690 ] DEBUG : Docker networks for project roottestparallelreplicasoverdistributed-gw5 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 19:57:34.767000 [ 690 ] DEBUG : Docker containers for project roottestparallelreplicasoverdistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 19:57:34.809000 [ 690 ] DEBUG : Docker volumes for project roottestparallelreplicasoverdistributed-gw5 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 19:57:34.809000 [ 690 ] DEBUG : Command:[docker container list --all --filter name='^/roottestparallelreplicasoverdistributed-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 19:57:34.838000 [ 690 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 19:57:34.838000 [ 690 ] DEBUG : No running containers for project: roottestparallelreplicasoverdistributed-gw5 (cluster.py:879, cleanup) 2026-10-29 19:57:34.838000 [ 690 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 19:57:34.861000 [ 690 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 19:57:34.861000 [ 690 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:57:34.912000 [ 690 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:57:34.912000 [ 690 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 19:57:34.912000 [ 690 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 19:57:34.912000 [ 690 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 19:57:34.934000 [ 690 ] DEBUG : Stdout:10 (cluster.py:145, run_and_check) 2026-10-29 19:57:34.934000 [ 690 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:57:34.971000 [ 690 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:57:34.971000 [ 690 ] DEBUG : Volumes pruned: 10 (cluster.py:915, cleanup) _____________________________________ test _____________________________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test(started_cluster): NODES["node2"].replace_config( "/etc/clickhouse-server/users.d/users.xml", config.format(sleep_in_send_data_ms=1000000), ) if NODES["node1"].is_built_with_thread_sanitizer(): pytest.skip("Hedged requests don't work under Thread Sanitizer") attempts = 0 while attempts < 1000: setting = NODES["node2"].http_query( "SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms'" ) if int(setting) == 1000000: break time.sleep(0.1) attempts += 1 assert attempts < 1000 start = time.time() NODES["node1"].query_and_get_error( "SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=0, async_socket_for_remote=0;" ) end = time.time() assert end - start < 10 start = time.time() error = NODES["node1"].query_and_get_error( "SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=0, async_socket_for_remote=1;" ) end = time.time() assert end - start < 10 # Check that exception about timeout wasn't thrown from DB::ReadBufferFromPocoSocket::nextImpl(). assert error.find("DB::ReadBufferFromPocoSocket::nextImpl()") == -1 start = time.time() error = NODES["node1"].query_and_get_error( "SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=1, async_socket_for_remote=1;" ) end = time.time() > assert end - start < 10 E assert (1793303874.5993814 - 1793303858.8654423) < 10 test_secure_socket/test.py:101: AssertionError ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name: project_name:roottestsecuresocket-gw2. Added instance name:node1 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env', '--project-name', 'roottestsecuresocket-gw2', '--file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Cluster name: project_name:roottestsecuresocket-gw2. Added instance name:node1 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env', '--project-name', 'roottestsecuresocket-gw2', '--file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name: project_name:roottestsecuresocket-gw2. Added instance name:node2 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env', '--project-name', 'roottestsecuresocket-gw2', '--file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Cluster name: project_name:roottestsecuresocket-gw2. Added instance name:node2 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env', '--project-name', 'roottestsecuresocket-gw2', '--file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Running tests in /ClickHouse/tests/integration/test_secure_socket/test.py Running tests in /ClickHouse/tests/integration/test_secure_socket/test.py Cluster start called. is_up=False Cluster start called. is_up=False Docker networks for project roottestsecuresocket-gw2 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestsecuresocket-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestsecuresocket-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestsecuresocket-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestsecuresocket-gw2 are DRIVER VOLUME NAME Docker volumes for project roottestsecuresocket-gw2 are DRIVER VOLUME NAME Cleanup called Cleanup called Docker networks for project roottestsecuresocket-gw2 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestsecuresocket-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestsecuresocket-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestsecuresocket-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestsecuresocket-gw2 are DRIVER VOLUME NAME Docker volumes for project roottestsecuresocket-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestsecuresocket-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestsecuresocket-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestsecuresocket-gw2 No running containers for project: roottestsecuresocket-gw2 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:10 Stdout:10 Command:[docker volume prune -f] Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Volumes pruned: 10 Volumes pruned: 10 Setup directory for instance: node1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/remote_servers.xml', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.crt', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.key', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/dhparam.pem', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/ssl_conf.xml'] to /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/configs/config.d Copy custom test config files ['/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/remote_servers.xml', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.crt', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.key', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/dhparam.pem', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/ssl_conf.xml'] to /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/database Setup database dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/database Setup logs dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/logs Setup logs dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/remote_servers.xml', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.crt', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.key', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/dhparam.pem', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/ssl_conf.xml'] to /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/configs/config.d Copy custom test config files ['/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/remote_servers.xml', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.crt', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.key', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/dhparam.pem', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/ssl_conf.xml'] to /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/database Setup database dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/database Setup logs dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/logs Setup logs dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml pull] Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml pull] Stderr: node2 Skipped - Image is already being pulled by node1 Stderr: node2 Skipped - Image is already being pulled by node1 Stderr: node1 Pulling Stderr: node1 Pulling Stderr: node1 Pulled Stderr: node1 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml up -d --no-recreate') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml up -d --no-recreate] Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml up -d --no-recreate] Stderr: Network roottestsecuresocket-gw2_default Creating Stderr: Network roottestsecuresocket-gw2_default Creating Stderr: Network roottestsecuresocket-gw2_default Created Stderr: Network roottestsecuresocket-gw2_default Created Stderr: Container roottestsecuresocket-gw2-node2-1 Creating Stderr: Container roottestsecuresocket-gw2-node2-1 Creating Stderr: Container roottestsecuresocket-gw2-node1-1 Creating Stderr: Container roottestsecuresocket-gw2-node1-1 Creating Stderr: Container roottestsecuresocket-gw2-node2-1 Created Stderr: Container roottestsecuresocket-gw2-node2-1 Created Stderr: Container roottestsecuresocket-gw2-node1-1 Created Stderr: Container roottestsecuresocket-gw2-node1-1 Created Stderr: Container roottestsecuresocket-gw2-node1-1 Starting Stderr: Container roottestsecuresocket-gw2-node1-1 Starting Stderr: Container roottestsecuresocket-gw2-node2-1 Starting Stderr: Container roottestsecuresocket-gw2-node2-1 Starting Stderr: Container roottestsecuresocket-gw2-node1-1 Started Stderr: Container roottestsecuresocket-gw2-node1-1 Started Stderr: Container roottestsecuresocket-gw2-node2-1 Started Stderr: Container roottestsecuresocket-gw2-node2-1 Started ClickHouse instance created ClickHouse instance created get_instance_ip instance_name=node1 get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.5.2... Waiting for ClickHouse start in node1, ip: 172.16.5.2... http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None ClickHouse node1 started ClickHouse node1 started get_instance_ip instance_name=node2 get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.5.3... Waiting for ClickHouse start in node2, ip: 172.16.5.3... http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0f79e33ca6e794c540ab838a48c920a4e67f69e734078bb342ece259f86424e3/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0f79e33ca6e794c540ab838a48c920a4e67f69e734078bb342ece259f86424e3/json HTTP/1.1" 200 None ClickHouse node2 started ClickHouse node2 started Executing query CREATE TABLE base_table (x UInt64) ENGINE = MergeTree ORDER BY x; on node2 Executing query CREATE TABLE base_table (x UInt64) ENGINE = MergeTree ORDER BY x; on node2 Executing query INSERT INTO base_table VALUES (5); on node2 Executing query INSERT INTO base_table VALUES (5); on node2 Executing query CREATE TABLE distributed_table (x UInt64) ENGINE = Distributed(test_cluster, default, base_table); on node1 Executing query CREATE TABLE distributed_table (x UInt64) ENGINE = Distributed(test_cluster, default, base_table); on node1 ------------------------------ Captured log setup ------------------------------ 2026-10-29 19:56:40.492000 [ 681 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1662, add_instance) 2026-10-29 19:56:40.493000 [ 681 ] DEBUG : Cluster name: project_name:roottestsecuresocket-gw2. Added instance name:node1 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env', '--project-name', 'roottestsecuresocket-gw2', '--file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:1948, add_instance) 2026-10-29 19:56:40.493000 [ 681 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1662, add_instance) 2026-10-29 19:56:40.494000 [ 681 ] DEBUG : Cluster name: project_name:roottestsecuresocket-gw2. Added instance name:node2 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env', '--project-name', 'roottestsecuresocket-gw2', '--file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:1948, add_instance) 2026-10-29 19:56:40.494000 [ 681 ] INFO : Running tests in /ClickHouse/tests/integration/test_secure_socket/test.py (cluster.py:2738, start) 2026-10-29 19:56:40.494000 [ 681 ] DEBUG : Cluster start called. is_up=False (cluster.py:2745, start) 2026-10-29 19:56:40.652000 [ 681 ] DEBUG : Docker networks for project roottestsecuresocket-gw2 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 19:56:40.777000 [ 681 ] DEBUG : Docker containers for project roottestsecuresocket-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 19:56:40.899000 [ 681 ] DEBUG : Docker volumes for project roottestsecuresocket-gw2 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 19:56:40.899000 [ 681 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 19:56:41.015000 [ 681 ] DEBUG : Docker networks for project roottestsecuresocket-gw2 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 19:56:41.089000 [ 681 ] DEBUG : Docker containers for project roottestsecuresocket-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 19:56:41.215000 [ 681 ] DEBUG : Docker volumes for project roottestsecuresocket-gw2 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 19:56:41.215000 [ 681 ] DEBUG : Command:[docker container list --all --filter name='^/roottestsecuresocket-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 19:56:41.271000 [ 681 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 19:56:41.271000 [ 681 ] DEBUG : No running containers for project: roottestsecuresocket-gw2 (cluster.py:879, cleanup) 2026-10-29 19:56:41.271000 [ 681 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 19:56:41.370000 [ 681 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 19:56:41.370000 [ 681 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:56:41.406000 [ 681 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:56:41.406000 [ 681 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 19:56:41.406000 [ 681 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 19:56:41.406000 [ 681 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 19:56:41.491000 [ 681 ] DEBUG : Stdout:10 (cluster.py:145, run_and_check) 2026-10-29 19:56:41.491000 [ 681 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:56:41.623000 [ 681 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:56:41.623000 [ 681 ] DEBUG : Volumes pruned: 10 (cluster.py:915, cleanup) 2026-10-29 19:56:41.623000 [ 681 ] DEBUG : Setup directory for instance: node1 (cluster.py:2758, start) 2026-10-29 19:56:41.624000 [ 681 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:56:41.624000 [ 681 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:56:41.624000 [ 681 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:56:41.626000 [ 681 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:56:41.627000 [ 681 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/remote_servers.xml', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.crt', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.key', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/dhparam.pem', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/ssl_conf.xml'] to /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:56:41.646000 [ 681 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/database (cluster.py:4758, create_dir) 2026-10-29 19:56:41.647000 [ 681 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/logs (cluster.py:4769, create_dir) 2026-10-29 19:56:41.647000 [ 681 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:56:41.647000 [ 681 ] DEBUG : Setup directory for instance: node2 (cluster.py:2758, start) 2026-10-29 19:56:41.647000 [ 681 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 19:56:41.647000 [ 681 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 19:56:41.648000 [ 681 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 19:56:41.648000 [ 681 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 19:56:41.648000 [ 681 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/remote_servers.xml', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.crt', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/server.key', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/dhparam.pem', '/ClickHouse/tests/integration/test_secure_socket/configs_secure/config.d/ssl_conf.xml'] to /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 19:56:41.699000 [ 681 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/database (cluster.py:4758, create_dir) 2026-10-29 19:56:41.699000 [ 681 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/logs (cluster.py:4769, create_dir) 2026-10-29 19:56:41.699000 [ 681 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-10-29 19:56:41.699000 [ 681 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env (cluster.py:96, _create_env_file) 2026-10-29 19:56:41.718000 [ 681 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 19:56:41.719000 [ 681 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 19:56:41.719000 [ 681 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 19:56:41.719000 [ 681 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 19:56:41.741000 [ 681 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2026-10-29 19:56:41.743000 [ 681 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-10-29 19:56:58.279000 [ 681 ] DEBUG : Stderr: node2 Skipped - Image is already being pulled by node1 (cluster.py:147, run_and_check) 2026-10-29 19:56:58.284000 [ 681 ] DEBUG : Stderr: node1 Pulling (cluster.py:147, run_and_check) 2026-10-29 19:56:58.284000 [ 681 ] DEBUG : Stderr: node1 Pulled (cluster.py:147, run_and_check) 2026-10-29 19:56:58.285000 [ 681 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml up -d --no-recreate') (cluster.py:3139, start) 2026-10-29 19:56:58.285000 [ 681 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml up -d --no-recreate] (cluster.py:121, run_and_check) 2026-10-29 19:57:14.020000 [ 681 ] DEBUG : Stderr: Network roottestsecuresocket-gw2_default Creating (cluster.py:147, run_and_check) 2026-10-29 19:57:14.043000 [ 681 ] DEBUG : Stderr: Network roottestsecuresocket-gw2_default Created (cluster.py:147, run_and_check) 2026-10-29 19:57:14.043000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node2-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:57:14.043000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node1-1 Creating (cluster.py:147, run_and_check) 2026-10-29 19:57:14.043000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node2-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:57:14.044000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node1-1 Created (cluster.py:147, run_and_check) 2026-10-29 19:57:14.044000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node1-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:57:14.044000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node2-1 Starting (cluster.py:147, run_and_check) 2026-10-29 19:57:14.044000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node1-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:57:14.044000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node2-1 Started (cluster.py:147, run_and_check) 2026-10-29 19:57:14.044000 [ 681 ] DEBUG : ClickHouse instance created (cluster.py:3147, start) 2026-10-29 19:57:14.045000 [ 681 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2005, get_instance_ip) 2026-10-29 19:57:14.087000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.089000 [ 681 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:57:14.095000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.096000 [ 681 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.5.2... (cluster.py:3155, start) 2026-10-29 19:57:14.100000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.105000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.213000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.320000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.429000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.534000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.636000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.740000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.877000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:14.980000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:15.084000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:15.190000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:15.305000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:15.423000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:15.535000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:15.643000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:15.771000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:15.891000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:16.001000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:16.109000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:16.236000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:16.344000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:16.456000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:16.564000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:16.707000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:16.825000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:16.948000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:17.069000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:17.191000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:17.294000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:17.398000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:17.519000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:17.719000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:17.825000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:17.932000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:18.041000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:18.151000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:18.264000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:18.368000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:18.475000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:18.583000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:18.688000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:18.791000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:18.902000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:19.006000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:19.115000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:19.259000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:19.374000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:19.483000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:19.586000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:19.695000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:19.801000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:19.904000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:20.007000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:20.111000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:20.214000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:20.320000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:20.424000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:20.529000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:20.655000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:20.762000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:20.870000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:20.990000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:21.102000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:21.217000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:21.415000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:22.613000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:22.736000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:22.843000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:22.949000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.056000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.176000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.283000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.387000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.489000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.595000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.698000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.804000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/095db2e149103f64bdb01180872847bd23646eb96f9ccd48abe6feea18c65866/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.805000 [ 681 ] DEBUG : ClickHouse node1 started (cluster.py:3159, start) 2026-10-29 19:57:23.805000 [ 681 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2005, get_instance_ip) 2026-10-29 19:57:23.811000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.811000 [ 681 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 19:57:23.814000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.814000 [ 681 ] DEBUG : Waiting for ClickHouse start in node2, ip: 172.16.5.3... (cluster.py:3155, start) 2026-10-29 19:57:23.816000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestsecuresocket-gw2-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.818000 [ 681 ] DEBUG : http://localhost:None "GET /v1.46/containers/0f79e33ca6e794c540ab838a48c920a4e67f69e734078bb342ece259f86424e3/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:23.818000 [ 681 ] DEBUG : ClickHouse node2 started (cluster.py:3159, start) 2026-10-29 19:57:23.834000 [ 681 ] DEBUG : Executing query CREATE TABLE base_table (x UInt64) ENGINE = MergeTree ORDER BY x; on node2 (cluster.py:3648, query) 2026-10-29 19:57:24.531000 [ 681 ] DEBUG : Executing query INSERT INTO base_table VALUES (5); on node2 (cluster.py:3648, query) 2026-10-29 19:57:25.010000 [ 681 ] DEBUG : Executing query CREATE TABLE distributed_table (x UInt64) ENGINE = Distributed(test_cluster, default, base_table); on node1 (cluster.py:3648, query) ----------------------------- Captured stderr call ----------------------------- run container_id:roottestsecuresocket-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 1000000\n \n \n' > /etc/clickhouse-server/users.d/users.xml"] run container_id:roottestsecuresocket-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 1000000\n \n \n' > /etc/clickhouse-server/users.d/users.xml"] Command:[docker exec roottestsecuresocket-gw2-node2-1 bash -c echo ' 1000000 ' > /etc/clickhouse-server/users.d/users.xml] Command:[docker exec roottestsecuresocket-gw2-node2-1 bash -c echo ' 1000000 ' > /etc/clickhouse-server/users.d/users.xml] Executing query SELECT value FROM system.build_options WHERE name = 'CXX_FLAGS' on node1 Executing query SELECT value FROM system.build_options WHERE name = 'CXX_FLAGS' on node1 Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface Starting new HTTP connection (1): 172.16.5.3:8123 Starting new HTTP connection (1): 172.16.5.3:8123 http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None Executing query SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=0, async_socket_for_remote=0; on node1 Executing query SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=0, async_socket_for_remote=0; on node1 Executing query SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=0, async_socket_for_remote=1; on node1 Executing query SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=0, async_socket_for_remote=1; on node1 Executing query SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=1, async_socket_for_remote=1; on node1 Executing query SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=1, async_socket_for_remote=1; on node1 ------------------------------ Captured log call ------------------------------- 2026-10-29 19:57:25.496000 [ 681 ] DEBUG : run container_id:roottestsecuresocket-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 1000000\n \n \n' > /etc/clickhouse-server/users.d/users.xml"] (cluster.py:2051, exec_in_container) 2026-10-29 19:57:25.496000 [ 681 ] DEBUG : Command:[docker exec roottestsecuresocket-gw2-node2-1 bash -c echo ' 1000000 ' > /etc/clickhouse-server/users.d/users.xml] (cluster.py:121, run_and_check) 2026-10-29 19:57:25.923000 [ 681 ] DEBUG : Executing query SELECT value FROM system.build_options WHERE name = 'CXX_FLAGS' on node1 (cluster.py:3648, query) 2026-10-29 19:57:26.290000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:26.292000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:26.317000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:26.426000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:26.427000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:26.451000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:26.551000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:26.552000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:26.575000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:26.676000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:26.677000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:26.713000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:26.817000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:26.818000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:26.847000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:26.958000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:26.959000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:26.994000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:27.102000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:27.103000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:27.144000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:27.257000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:27.258000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:27.300000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:27.403000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:27.404000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:27.455000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:27.558000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:27.559000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:27.574000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:27.685000 [ 681 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node2 via HTTP interface (cluster.py:3899, http_query_and_get_answer_with_error) 2026-10-29 19:57:27.686000 [ 681 ] DEBUG : Starting new HTTP connection (1): 172.16.5.3:8123 (connectionpool.py:245, _new_conn) 2026-10-29 19:57:27.767000 [ 681 ] DEBUG : http://172.16.5.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 19:57:27.780000 [ 681 ] DEBUG : Executing query SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=0, async_socket_for_remote=0; on node1 (cluster.py:3731, query_and_get_error) 2026-10-29 19:57:33.247000 [ 681 ] DEBUG : Executing query SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=0, async_socket_for_remote=1; on node1 (cluster.py:3731, query_and_get_error) 2026-10-29 19:57:38.865000 [ 681 ] DEBUG : Executing query SELECT * FROM distributed_table settings receive_timeout=5, send_timeout=5, use_hedged_requests=1, async_socket_for_remote=1; on node1 (cluster.py:3731, query_and_get_error) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml stop --timeout 20] Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml stop --timeout 20] Stderr: Container roottestsecuresocket-gw2-node1-1 Stopping Stderr: Container roottestsecuresocket-gw2-node1-1 Stopping Stderr: Container roottestsecuresocket-gw2-node2-1 Stopping Stderr: Container roottestsecuresocket-gw2-node2-1 Stopping Stderr: Container roottestsecuresocket-gw2-node2-1 Stopped Stderr: Container roottestsecuresocket-gw2-node2-1 Stopped Stderr: Container roottestsecuresocket-gw2-node1-1 Stopped Stderr: Container roottestsecuresocket-gw2-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml down --volumes] Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml down --volumes] Stderr: Container roottestsecuresocket-gw2-node1-1 Stopping Stderr: Container roottestsecuresocket-gw2-node1-1 Stopping Stderr: Container roottestsecuresocket-gw2-node2-1 Stopping Stderr: Container roottestsecuresocket-gw2-node2-1 Stopping Stderr: Container roottestsecuresocket-gw2-node2-1 Stopped Stderr: Container roottestsecuresocket-gw2-node2-1 Stopped Stderr: Container roottestsecuresocket-gw2-node2-1 Removing Stderr: Container roottestsecuresocket-gw2-node2-1 Removing Stderr: Container roottestsecuresocket-gw2-node1-1 Stopped Stderr: Container roottestsecuresocket-gw2-node1-1 Stopped Stderr: Container roottestsecuresocket-gw2-node1-1 Removing Stderr: Container roottestsecuresocket-gw2-node1-1 Removing Stderr: Container roottestsecuresocket-gw2-node1-1 Removed Stderr: Container roottestsecuresocket-gw2-node1-1 Removed Stderr: Container roottestsecuresocket-gw2-node2-1 Removed Stderr: Container roottestsecuresocket-gw2-node2-1 Removed Stderr: Network roottestsecuresocket-gw2_default Removing Stderr: Network roottestsecuresocket-gw2_default Removing Stderr: Network roottestsecuresocket-gw2_default Removed Stderr: Network roottestsecuresocket-gw2_default Removed Cleanup called Cleanup called Docker networks for project roottestsecuresocket-gw2 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestsecuresocket-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestsecuresocket-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestsecuresocket-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestsecuresocket-gw2 are DRIVER VOLUME NAME Docker volumes for project roottestsecuresocket-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestsecuresocket-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestsecuresocket-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestsecuresocket-gw2 No running containers for project: roottestsecuresocket-gw2 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:8 Stdout:8 Command:[docker volume prune -f] Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Volumes pruned: 8 Volumes pruned: 8 ---------------------------- Captured log teardown ----------------------------- 2026-10-29 19:57:54.752000 [ 681 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml stop --timeout 20] (cluster.py:121, run_and_check) 2026-10-29 19:58:12.046000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:58:12.047000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:58:12.047000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:58:12.047000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:58:12.048000 [ 681 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:58:12.133000 [ 681 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 19:58:12.171000 [ 681 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/.env --project-name roottestsecuresocket-gw2 --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_secure_socket/_instances-0-gw2/node2/docker-compose.yml down --volumes] (cluster.py:121, run_and_check) 2026-10-29 19:58:14.499000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:58:14.499000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 19:58:14.499000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:58:14.499000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node2-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:58:14.499000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 19:58:14.499000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node1-1 Removing (cluster.py:147, run_and_check) 2026-10-29 19:58:14.499000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node1-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:58:14.499000 [ 681 ] DEBUG : Stderr: Container roottestsecuresocket-gw2-node2-1 Removed (cluster.py:147, run_and_check) 2026-10-29 19:58:14.499000 [ 681 ] DEBUG : Stderr: Network roottestsecuresocket-gw2_default Removing (cluster.py:147, run_and_check) 2026-10-29 19:58:14.499000 [ 681 ] DEBUG : Stderr: Network roottestsecuresocket-gw2_default Removed (cluster.py:147, run_and_check) 2026-10-29 19:58:14.500000 [ 681 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 19:58:14.583000 [ 681 ] DEBUG : Docker networks for project roottestsecuresocket-gw2 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 19:58:14.725000 [ 681 ] DEBUG : Docker containers for project roottestsecuresocket-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 19:58:14.862000 [ 681 ] DEBUG : Docker volumes for project roottestsecuresocket-gw2 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 19:58:14.863000 [ 681 ] DEBUG : Command:[docker container list --all --filter name='^/roottestsecuresocket-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 19:58:14.918000 [ 681 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 19:58:14.919000 [ 681 ] DEBUG : No running containers for project: roottestsecuresocket-gw2 (cluster.py:879, cleanup) 2026-10-29 19:58:14.919000 [ 681 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 19:58:15.227000 [ 681 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 19:58:15.227000 [ 681 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:58:15.413000 [ 681 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:58:15.413000 [ 681 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 19:58:15.414000 [ 681 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 19:58:15.414000 [ 681 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 19:58:15.573000 [ 681 ] DEBUG : Stdout:8 (cluster.py:145, run_and_check) 2026-10-29 19:58:15.573000 [ 681 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-10-29 19:58:15.663000 [ 681 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 19:58:15.663000 [ 681 ] DEBUG : Volumes pruned: 8 (cluster.py:915, cleanup) ___________________________ test_backup_outer_table ____________________________ [gw4] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = cleanup = None def test_backup_outer_table(started_cluster, cleanup): > do_test_backup(True) test_refreshable_mv/test.py:444: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_refreshable_mv/test.py:427: in do_test_backup node1.query(f"RESTORE ALL ON CLUSTER 'default' FROM {backup_destination};") helpers/cluster.py:3649: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 231, stderr: Received exception from server (version 25.3.14): E Code: 999. DB::Exception: Received from 172.16.6.9:9000. DB::Exception: Got error from host node1:9000. Coordination::Exception. Coordination::Exception: Coordination error: Connection loss, path /clickhouse/tables/265923bd-c82b-4f67-a396-3efb26a2f317/shard1/paused. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb E 3. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type, std::type_identity::type>, char const*&&, String const&) @ 0x0000000010ef6204 E 4. Coordination::Exception::fromPath(Coordination::Error, String const&) @ 0x0000000010ef541f E 5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:556: zkutil::ZooKeeper::tryRemove(String const&, int) @ 0x00000000315f2a96 E 6. ./build_docker/./src/Storages/MaterializedView/RefreshTask.cpp:316: DB::RefreshTask::startReplicated() @ 0x000000002fd41e60 E 7. ./build_docker/./src/Backups/RestorerFromBackup.cpp:1189: DB::RestorerFromBackup::finalizeTables() @ 0x00000000279cd374 E 8. ./build_docker/./src/Backups/RestorerFromBackup.cpp:180: DB::RestorerFromBackup::run(DB::RestorerFromBackup::Mode) @ 0x00000000279c3de5 E 9. ./build_docker/./src/Backups/BackupsWorker.cpp:979: DB::BackupsWorker::doRestore(std::shared_ptr const&, String const&, DB::BackupInfo const&, DB::RestoreSettings, std::shared_ptr, std::shared_ptr, std::shared_ptr const&, bool, std::shared_ptr const&) @ 0x000000002788efaf E 10. ./build_docker/./src/Backups/BackupsWorker.cpp:824: DB::BackupsWorker::RestoreStarter::doRestore() @ 0x00000000278a7580 E 11. ./build_docker/./src/Backups/BackupsWorker.cpp:871: void std::__function::__policy_invoker::__call_impl[abi:ne190107] const&, std::shared_ptr)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000027897d93 E 12. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000026e5fc5a E 13. ./contrib/llvm-project/libcxx/include/future:1589: std::packaged_task::operator()() @ 0x0000000026e6024c E 14. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c35ca92 E 15. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::ThreadFromThreadPool::*)(), ThreadPoolImpl>::ThreadFromThreadPool*>(void (ThreadPoolImpl>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl>::ThreadFromThreadPool*&&)::'lambda'()::operator()() @ 0x000000001c36a9e3 E 16. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c357271 E 17. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: void* std::__thread_proxy[abi:ne190107]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000001c3656d0 E 18. asan_thread_start(void*) @ 0x000000000c676e37 E 19. ? @ 0x00007fbb12279ac3 E 20. ? @ 0x00007fbb1230b850 E E Job's origin stack trace: E 0. ./build_docker/./src/Common/StackTrace.cpp:386: StackTrace::StackTrace() @ 0x000000001c1f3727 E 1. ./build_docker/./src/Common/ThreadPool.cpp:130: void boost::heap::priority_queue, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function&&, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000001c355824 E 2. ./build_docker/./src/Common/ThreadPool.cpp:401: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool) @ 0x000000001c36052a E 3. ./build_docker/./src/Common/ThreadPool.cpp:494: ThreadPoolImpl>::scheduleOrThrowOnError(std::function, Priority) @ 0x000000001c35f957 E 4. ./src/Common/threadPoolCallbackRunner.h:52: std::function (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority) @ 0x0000000026e5e63c E 5. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: std::future std::__function::__policy_invoker (std::function&&, Priority)>::__call_impl[abi:ne190107] (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority), std::future (std::function&&, Priority)>>(std::__function::__policy_storage const*, std::function&&, Priority&&) @ 0x0000000026e5e194 E 6. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000027883cb1 E 7. ./build_docker/./src/Backups/BackupsWorker.cpp:341: DB::BackupsWorker::start(std::shared_ptr const&, std::shared_ptr) @ 0x0000000027883072 E 8. ./build_docker/./src/Interpreters/InterpreterBackupQuery.cpp:44: DB::InterpreterBackupQuery::execute() @ 0x000000002b0379fe E 9. ./build_docker/./src/Interpreters/executeQuery.cpp:1457: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af36de3 E 10. ./build_docker/./src/Interpreters/executeQuery.cpp:1761: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr, std::function, DB::QueryFlags, std::optional const&, std::function const&, std::optional const&)>) @ 0x000000002af3e8ff E 11. ./build_docker/./src/Interpreters/DDLWorker.cpp:510: DB::DDLWorker::tryExecuteQuery(DB::DDLTaskBase&, std::shared_ptr const&, bool) @ 0x0000000029ac11ef E 12. ./build_docker/./src/Interpreters/DDLWorker.cpp:675: DB::DDLWorker::processTask(DB::DDLTaskBase&, std::shared_ptr const&, bool) @ 0x0000000029abce79 E 13. ./build_docker/./src/Interpreters/DDLWorker.cpp:453: DB::DDLWorker::scheduleTasks(bool) @ 0x0000000029ab8499 E 14. ./build_docker/./src/Interpreters/DDLWorker.cpp:1203: DB::DDLWorker::runMainThread() @ 0x0000000029aac66c E 15. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'()::operator()() @ 0x0000000029adfe03 E 16. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c357271 E 17. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: void* std::__thread_proxy[abi:ne190107]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000001c3656d0 E 18. asan_thread_start(void*) @ 0x000000000c676e37 E 19. ? @ 0x00007fbb12279ac3 E 20. ? @ 0x00007fbb1230b850 E . Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 E 2. ./src/Common/Exception.h:111: DB::readException(DB::ReadBuffer&, String const&, bool) @ 0x000000001c292ee5 E 3. ./build_docker/./src/Backups/BackupCoordinationStageSync.cpp:1450: DB::BackupCoordinationStageSync::parseErrorNode(String const&) const @ 0x00000000279725bc E 4. ./build_docker/./src/Backups/BackupCoordinationStageSync.cpp:584: DB::BackupCoordinationStageSync::readCurrentState(std::shared_ptr) @ 0x0000000027978104 E 5. ./build_docker/./src/Backups/BackupCoordinationStageSync.cpp:485: DB::BackupCoordinationStageSync::watchingThread() @ 0x0000000027974dd9 E 6. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000026e5fc5a E 7. ./contrib/llvm-project/libcxx/include/future:1589: std::packaged_task::operator()() @ 0x0000000026e6024c E 8. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c35ca92 E 9. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::ThreadFromThreadPool::*)(), ThreadPoolImpl>::ThreadFromThreadPool*>(void (ThreadPoolImpl>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl>::ThreadFromThreadPool*&&)::'lambda'()::operator()() @ 0x000000001c36a9e3 E 10. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c357271 E 11. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: void* std::__thread_proxy[abi:ne190107]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000001c3656d0 E 12. asan_thread_start(void*) @ 0x000000000c676e37 E 13. ? @ 0x00007fbb12279ac3 E 14. ? @ 0x00007fbb1230b850 E E Job's origin stack trace: E 0. ./build_docker/./src/Common/StackTrace.cpp:386: StackTrace::StackTrace() @ 0x000000001c1f3727 E 1. ./build_docker/./src/Common/ThreadPool.cpp:130: void boost::heap::priority_queue, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function&&, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000001c355824 E 2. ./build_docker/./src/Common/ThreadPool.cpp:401: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool) @ 0x000000001c36052a E 3. ./build_docker/./src/Common/ThreadPool.cpp:494: ThreadPoolImpl>::scheduleOrThrowOnError(std::function, Priority) @ 0x000000001c35f957 E 4. ./src/Common/threadPoolCallbackRunner.h:52: std::function (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority) @ 0x0000000026e5e63c E 5. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: std::future std::__function::__policy_invoker (std::function&&, Priority)>::__call_impl[abi:ne190107] (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority), std::future (std::function&&, Priority)>>(std::__function::__policy_storage const*, std::function&&, Priority&&) @ 0x0000000026e5e194 E 6. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000002796bcf8 E 7. ./build_docker/./src/Backups/BackupCoordinationStageSync.cpp:137: DB::BackupCoordinationStageSync::BackupCoordinationStageSync(bool, String const&, String const&, std::vector> const&, bool, DB::BackupConcurrencyCounters&, DB::WithRetries const&, std::function (std::function&&, Priority)>, std::shared_ptr, std::shared_ptr) @ 0x0000000027967d26 E 8. ./build_docker/./src/Backups/RestoreCoordinationOnCluster.cpp:39: DB::RestoreCoordinationOnCluster::RestoreCoordinationOnCluster(StrongTypedef, DB::UUIDTag> const&, String const&, std::function ()>, DB::BackupKeeperSettings const&, String const&, std::vector> const&, bool, DB::BackupConcurrencyCounters&, std::function (std::function&&, Priority)>, std::shared_ptr) @ 0x00000000279a638b E 9. ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:41: DB::BackupsWorker::makeRestoreCoordination(bool, DB::RestoreSettings const&, std::shared_ptr const&) const @ 0x0000000027891680 E 10. ./build_docker/./src/Backups/BackupsWorker.cpp:822: DB::BackupsWorker::RestoreStarter::doRestore() @ 0x00000000278a72e0 E 11. ./build_docker/./src/Backups/BackupsWorker.cpp:871: void std::__function::__policy_invoker::__call_impl[abi:ne190107] const&, std::shared_ptr)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000027897d93 E 12. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000026e5fc5a E 13. ./contrib/llvm-project/libcxx/include/future:1589: std::packaged_task::operator()() @ 0x0000000026e6024c E 14. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c35ca92 E 15. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::ThreadFromThreadPool::*)(), ThreadPoolImpl>::ThreadFromThreadPool*>(void (ThreadPoolImpl>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl>::ThreadFromThreadPool*&&)::'lambda'()::operator()() @ 0x000000001c36a9e3 E 16. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c357271 E 17. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: void* std::__thread_proxy[abi:ne190107]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000001c3656d0 E 18. asan_thread_start(void*) @ 0x000000000c676e37 E 19. ? @ 0x00007fbb12279ac3 E 20. ? @ 0x00007fbb1230b850 E E Job's origin stack trace: E 0. ./build_docker/./src/Common/StackTrace.cpp:386: StackTrace::StackTrace() @ 0x000000001c1f3727 E 1. ./build_docker/./src/Common/ThreadPool.cpp:130: void boost::heap::priority_queue, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function&&, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000001c355824 E 2. ./build_docker/./src/Common/ThreadPool.cpp:401: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool) @ 0x000000001c36052a E 3. ./build_docker/./src/Common/ThreadPool.cpp:494: ThreadPoolImpl>::scheduleOrThrowOnError(std::function, Priority) @ 0x000000001c35f957 E 4. ./src/Common/threadPoolCallbackRunner.h:52: std::function (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority) @ 0x0000000026e5e63c E 5. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: std::future std::__function::__policy_invoker (std::function&&, Priority)>::__call_impl[abi:ne190107] (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority), std::future (std::function&&, Priority)>>(std::__function::__policy_storage const*, std::function&&, Priority&&) @ 0x0000000026e5e194 E 6. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000027883cb1 E 7. ./build_docker/./src/Backups/BackupsWorker.cpp:341: DB::BackupsWorker::start(std::shared_ptr const&, std::shared_ptr) @ 0x0000000027883072 E 8. ./build_docker/./src/Interpreters/InterpreterBackupQuery.cpp:44: DB::InterpreterBackupQuery::execute() @ 0x000000002b0379fe E 9. ./build_docker/./src/Interpreters/executeQuery.cpp:1457: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af36de3 E 10. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 E 11. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 E 12. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa E 13. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f E 14. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 E 15. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b E 16. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 E 17. asan_thread_start(void*) @ 0x000000000c676e37 E 18. ? @ 0x00007fbb12279ac3 E 19. ? @ 0x00007fbb1230b850 E . (KEEPER_EXCEPTION) E (query: RESTORE ALL ON CLUSTER 'default' FROM S3('http://minio1:9001/root/data/backups/backup2', 'minio', 'ClickHouse_Minio_P@ssw0rd');) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node1 Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node1 Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node2 Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node2 Executing query create table re.tgt (x Int64) engine ReplicatedMergeTree order by x on node1 Executing query create table re.tgt (x Int64) engine ReplicatedMergeTree order by x on node1 Executing query create table re.src (x Int64) engine ReplicatedMergeTree order by x;insert into re.src values (1); on node1 Executing query create table re.src (x Int64) engine ReplicatedMergeTree order by x;insert into re.src values (1); on node1 Executing query system sync database replica re;system sync replica re.src;create materialized view re.rmv refresh every 1 second TO re.tgt as select x from re.src;system wait view re.rmv on node2 Executing query system sync database replica re;system sync replica re.src;create materialized view re.rmv refresh every 1 second TO re.tgt as select x from re.src;system wait view re.rmv on node2 Executing query select * from re.tgt on node2 Executing query select * from re.tgt on node2 Executing query BACKUP ALL EXCEPT DATABASE system ON CLUSTER 'default' TO S3('http://minio1:9001/root/data/backups/backup2', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 Executing query BACKUP ALL EXCEPT DATABASE system ON CLUSTER 'default' TO S3('http://minio1:9001/root/data/backups/backup2', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node1 Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node1 Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node2 Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node2 Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node1 Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node1 Executing query RESTORE ALL ON CLUSTER 'default' FROM S3('http://minio1:9001/root/data/backups/backup2', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 Executing query RESTORE ALL ON CLUSTER 'default' FROM S3('http://minio1:9001/root/data/backups/backup2', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 ------------------------------ Captured log call ------------------------------- 2026-10-29 19:57:26.242000 [ 687 ] DEBUG : Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node1 (cluster.py:3648, query) 2026-10-29 19:57:26.623000 [ 687 ] DEBUG : Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node2 (cluster.py:3648, query) 2026-10-29 19:57:27.099000 [ 687 ] DEBUG : Executing query create table re.tgt (x Int64) engine ReplicatedMergeTree order by x on node1 (cluster.py:3648, query) 2026-10-29 19:57:28.087000 [ 687 ] DEBUG : Executing query create table re.src (x Int64) engine ReplicatedMergeTree order by x;insert into re.src values (1); on node1 (cluster.py:3648, query) 2026-10-29 19:57:31.909000 [ 687 ] DEBUG : Executing query system sync database replica re;system sync replica re.src;create materialized view re.rmv refresh every 1 second TO re.tgt as select x from re.src;system wait view re.rmv on node2 (cluster.py:3648, query) 2026-10-29 19:57:33.251000 [ 687 ] DEBUG : Executing query select * from re.tgt on node2 (cluster.py:3648, query) 2026-10-29 19:57:33.825000 [ 687 ] DEBUG : Executing query BACKUP ALL EXCEPT DATABASE system ON CLUSTER 'default' TO S3('http://minio1:9001/root/data/backups/backup2', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 (cluster.py:3648, query) 2026-10-29 19:57:36.056000 [ 687 ] DEBUG : Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node1 (cluster.py:3648, query) 2026-10-29 19:57:36.536000 [ 687 ] DEBUG : Executing query drop database re sync;drop database if exists re_broken_replicated_tables sync; on node2 (cluster.py:3648, query) 2026-10-29 19:57:37.414000 [ 687 ] DEBUG : Executing query SELECT count() FROM system.tables where database='re' AND name in ('src', 'rmv') on node1 (cluster.py:3648, query) 2026-10-29 19:57:37.733000 [ 687 ] DEBUG : Executing query RESTORE ALL ON CLUSTER 'default' FROM S3('http://minio1:9001/root/data/backups/backup2', 'minio', 'ClickHouse_Minio_P@ssw0rd'); on node1 (cluster.py:3648, query) --------------------------- Captured stderr teardown --------------------------- Executing query drop database if exists re sync;drop table if exists system.a; on node1 Executing query drop database if exists re sync;drop table if exists system.a; on node1 Executing query drop database if exists re sync;drop table if exists system.a; on node2 Executing query drop database if exists re sync;drop table if exists system.a; on node2 Executing query drop database if exists re sync;drop table if exists system.a; on reading_node Executing query drop database if exists re sync;drop table if exists system.a; on reading_node ---------------------------- Captured log teardown ----------------------------- 2026-10-29 19:58:02.110000 [ 687 ] DEBUG : Executing query drop database if exists re sync;drop table if exists system.a; on node1 (cluster.py:3648, query) 2026-10-29 19:58:06.169000 [ 687 ] DEBUG : Executing query drop database if exists re sync;drop table if exists system.a; on node2 (cluster.py:3648, query) 2026-10-29 19:58:06.915000 [ 687 ] DEBUG : Executing query drop database if exists re sync;drop table if exists system.a; on reading_node (cluster.py:3648, query) __________________________________ test_pause __________________________________ [gw4] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = cleanup = None def test_pause(started_cluster, cleanup): for node in nodes: > node.query( "create database re engine = Replicated('/test/re', 'shard1', '{replica}');" ) test_refreshable_mv/test.py:347: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3649: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 231, stderr: Received exception from server (version 25.3.14): E Code: 999. DB::Exception: Received from 172.16.6.9:9000. Coordination::Exception. Coordination::Exception: Coordination error: Connection loss, path /test/re. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x000000003867ed11 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001c0899d1 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000c6c41cb E 3. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type, std::type_identity::type>, char const*&&, String const&) @ 0x0000000010ef6204 E 4. Coordination::Exception::fromPath(Coordination::Error, String const&) @ 0x0000000010ef541f E 5. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:599: zkutil::ZooKeeper::existsWatch(String const&, Coordination::Stat*, std::function) @ 0x00000000315f3d38 E 6. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:580: zkutil::ZooKeeper::exists(String const&, Coordination::Stat*, std::shared_ptr const&) @ 0x00000000315ee1f5 E 7. ./build_docker/./src/Databases/DatabaseReplicated.cpp:487: DB::DatabaseReplicated::tryConnectToZooKeeperAndInitDatabase(DB::LoadingStrictnessLevel) @ 0x00000000285574c4 E 8. ./build_docker/./src/Databases/DatabaseReplicated.cpp:783: DB::DatabaseReplicated::beforeLoadingMetadata(std::shared_ptr, DB::LoadingStrictnessLevel) @ 0x0000000028566cd1 E 9. ./build_docker/./src/Databases/TablesLoader.cpp:57: DB::TablesLoader::loadTablesAsync(std::unordered_set, std::hash>, std::equal_to>, std::allocator>>) @ 0x000000002a6aeb80 E 10. ./build_docker/./src/Interpreters/InterpreterCreateQuery.cpp:383: DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x000000002a65810e E 11. ./build_docker/./src/Interpreters/InterpreterCreateQuery.cpp:2354: DB::InterpreterCreateQuery::execute() @ 0x000000002a6963fb E 12. ./build_docker/./src/Interpreters/executeQuery.cpp:1457: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000002af36de3 E 13. ./build_docker/./src/Interpreters/executeQuery.cpp:1624: DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000002af2ecc5 E 14. ./build_docker/./src/Server/TCPHandler.cpp:664: DB::TCPHandler::runImpl() @ 0x00000000300bf736 E 15. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x00000000300ff3fa E 16. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x000000003885a00f E 17. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000003885acd7 E 18. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x000000003876962b E 19. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000038763228 E 20. asan_thread_start(void*) @ 0x000000000c676e37 E 21. ? @ 0x00007fbb12279ac3 E 22. ? @ 0x00007fbb1230b850 E . (KEEPER_EXCEPTION) E (query: create database re engine = Replicated('/test/re', 'shard1', '{replica}');) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node1 Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node1 ------------------------------ Captured log call ------------------------------- 2026-10-29 19:58:11.817000 [ 687 ] DEBUG : Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node1 (cluster.py:3648, query) --------------------------- Captured stderr teardown --------------------------- Executing query drop database if exists re sync;drop table if exists system.a; on node1 Executing query drop database if exists re sync;drop table if exists system.a; on node1 Executing query drop database if exists re sync;drop table if exists system.a; on node2 Executing query drop database if exists re sync;drop table if exists system.a; on node2 Executing query drop database if exists re sync;drop table if exists system.a; on reading_node Executing query drop database if exists re sync;drop table if exists system.a; on reading_node ---------------------------- Captured log teardown ----------------------------- 2026-10-29 19:58:12.847000 [ 687 ] DEBUG : Executing query drop database if exists re sync;drop table if exists system.a; on node1 (cluster.py:3648, query) 2026-10-29 19:58:13.420000 [ 687 ] DEBUG : Executing query drop database if exists re sync;drop table if exists system.a; on node2 (cluster.py:3648, query) 2026-10-29 19:58:14.427000 [ 687 ] DEBUG : Executing query drop database if exists re sync;drop table if exists system.a; on reading_node (cluster.py:3648, query) ----------------- generated report log file: parallel2_0.jsonl ----------------- ============================== slowest durations =============================== 600.84s call test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss 249.02s call test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table 138.36s call test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke 131.21s call test_replicated_user_defined_functions/test.py::test_reload_zookeeper 120.17s call test_recompression_ttl/test.py::test_recompression_simple 112.52s setup test_settings_constraints_distributed/test.py::test_insert_clamps_settings 99.66s call test_remove_stale_moving_parts/test.py::test_remove_stale_moving_parts_without_zookeeper 88.96s teardown test_refreshable_mv/test.py::test_backup_inner_table 81.90s call test_refreshable_mv/test.py::test_backup_inner_table 78.86s setup test_replicated_engine_arguments/test.py::test_replicated_engine_with_arguments 76.10s call test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] 74.07s call test_recompression_ttl/test.py::test_recompression_multiple_ttls 62.67s teardown test_s3_cluster/test.py::test_wrong_cluster 57.85s call test_storage_iceberg/test.py::test_cluster_table_function[s3-1] 56.01s call test_storage_iceberg/test.py::test_cluster_table_function[s3-2] 53.80s call test_replicated_user_defined_functions/test.py::test_replication 49.64s setup test_remove_stale_moving_parts/test.py::test_remove_stale_moving_parts_without_zookeeper 48.14s setup test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility 47.11s call test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db 46.05s setup test_replicated_database_alter_modify_order_by/test.py::test_alter_modify_order_by 46.03s call test_replicated_user_defined_functions/test.py::test_start_without_zookeeper 45.01s setup test_secure_socket/test.py::test 44.65s setup test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] 44.27s call test_replicated_user_defined_functions/test.py::test_server_restart 40.38s call test_storage_iceberg/test.py::test_cluster_table_function[azure-1] 38.31s teardown test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys 36.68s call test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility 34.74s call test_refreshable_mv/test.py::test_backup_outer_table 34.64s call test_storage_iceberg/test.py::test_cluster_table_function[azure-2] 32.47s teardown test_remove_stale_moving_parts/test.py::test_remove_stale_moving_parts_without_zookeeper 31.96s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6] 29.53s setup test_settings_constraints_distributed/test.py::test_select_clamps_settings 29.52s teardown test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] 29.12s call test_secure_socket/test.py::test 28.81s call test_storage_iceberg/test.py::test_delete_files[azure-1] 28.42s call test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-1] 28.29s call test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-2] 27.65s teardown test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-2] 26.77s teardown test_recompression_ttl/test.py::test_recompression_simple 26.68s setup test_storage_delta/test_imds.py::test_session_token 26.38s teardown test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] 24.81s call test_s3_zero_copy_ttl/test.py::test_ttl_move_and_s3 24.52s setup test_s3_cluster/test.py::test_ambiguous_join 24.48s teardown test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage 24.48s teardown test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore 24.24s setup test_refreshable_mv/test.py::test_adding_replica 23.41s call test_recompression_ttl/test.py::test_recompression_replicated 23.33s setup test_storage_iceberg/test.py::test_cluster_table_function[azure-1] 22.81s setup test_render_log_file_name_templates/test.py::test_check_file_names 22.62s teardown test_refreshable_mv/test.py::test_refreshable_mv_in_system_db 22.49s setup test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys 22.49s setup test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table 21.95s teardown test_s3_zero_copy_ttl/test.py::test_ttl_move_and_s3 21.73s teardown test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss 21.72s teardown test_storage_delta/test_imds.py::test_session_token 21.18s setup test_replicated_user_defined_functions/test.py::test_create_and_drop 20.93s teardown test_secure_socket/test.py::test 20.33s call test_settings_constraints_distributed/test.py::test_select_clamps_settings 20.29s call test_replicated_engine_arguments/test.py::test_replicated_engine_with_arguments 19.95s call test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys 19.75s setup test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss 19.02s setup test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1] 18.24s teardown test_shard_names/test.py::test_shard_names 18.12s call test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-1] 17.96s setup test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] 17.90s setup test_shard_names/test.py::test_incorrect_shard_names 17.56s setup test_recompression_ttl/test.py::test_recompression_multiple_ttls 17.40s call test_storage_iceberg/test.py::test_delete_files[azure-2] 17.19s teardown test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility 16.83s setup test_shard_level_const_function/test.py::test_remote 16.68s teardown test_replicated_engine_arguments/test.py::test_replicated_engine_with_arguments 15.43s setup test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage 14.71s call test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 14.66s setup test_ssh/test.py::test_create_table 14.59s setup test_settings_randomization/test.py::test_settings_randomization 14.34s setup test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 13.44s call test_storage_delta/test_imds.py::test_session_token 13.13s call test_refreshable_mv/test.py::test_refreshable_mv_in_system_db 12.19s setup test_s3_zero_copy_ttl/test.py::test_ttl_move_and_s3 11.72s call test_s3_cluster/test.py::test_cluster_format_detection 11.47s call test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys 10.68s call test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6] 10.10s call test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4] 10.04s call test_storage_iceberg/test.py::test_delete_files[local-2] 10.00s call test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-1] 9.66s teardown test_refreshable_mv/test.py::test_backup_outer_table 9.46s call test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-2] 9.24s call test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-1] 9.11s call test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-2] 9.00s call test_storage_iceberg/test.py::test_delete_files[local-1] 8.80s call test_s3_cluster/test.py::test_cluster_default_expression 8.44s call test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3] 8.40s call test_storage_iceberg/test.py::test_evolved_schema_complex[local-2] 8.37s call test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-2] 8.23s call test_settings_constraints_distributed/test.py::test_insert_clamps_settings 7.94s call test_s3_cluster/test.py::test_cluster_with_header 7.81s teardown test_shard_level_const_function/test.py::test_remote 7.51s call test_refreshable_mv/test.py::test_refreshable_mv_in_read_only_node 7.23s call test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5] 7.07s call test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 6.58s call test_s3_cluster/test.py::test_select_all 6.44s setup test_shutdown_static_destructor_failure/test.py::test_shutdown 6.44s call test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore 6.38s call test_s3_cluster/test.py::test_count 6.35s call test_storage_iceberg/test.py::test_delete_files[s3-1] 6.22s call test_shard_names/test.py::test_incorrect_shard_names 6.00s call test_storage_iceberg/test.py::test_delete_files[s3-2] 5.89s call test_refreshable_mv/test.py::test_adding_replica 5.86s teardown test_settings_constraints_distributed/test.py::test_select_clamps_settings 5.66s call test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] 5.16s call test_storage_iceberg/test.py::test_evolved_schema_complex[local-1] 5.03s teardown test_replicated_user_defined_functions/test.py::test_start_without_zookeeper 4.83s call test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2] 4.41s call test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1] 4.38s call test_replicated_database_alter_modify_order_by/test.py::test_alter_modify_order_by 4.37s call test_replicated_user_defined_functions/test.py::test_replication_replace_by_another_node_after_creation 4.23s teardown test_render_log_file_name_templates/test.py::test_check_file_names 4.05s call test_s3_cluster/test.py::test_count_macro 3.87s teardown test_settings_randomization/test.py::test_settings_randomization 3.71s teardown test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] 3.63s teardown test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 3.62s call test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] 3.44s call test_shard_names/test.py::test_shard_names 3.42s teardown test_ssh/test.py::test_simple_query_with_paramiko 3.28s call test_storage_iceberg/test.py::test_evolved_schema_complex[azure-1] 3.22s teardown test_replicated_database_alter_modify_order_by/test.py::test_alter_modify_order_by 3.09s call test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] 2.92s call test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] 2.92s teardown test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] 2.68s call test_s3_cluster/test.py::test_union_all 2.68s teardown test_refreshable_mv/test.py::test_pause 2.66s call test_replicated_user_defined_functions/test.py::test_create_and_replace 2.42s call test_shutdown_static_destructor_failure/test.py::test_shutdown 2.40s call test_s3_cluster/test.py::test_cluster_with_named_collection 2.36s call test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] 2.34s teardown test_refreshable_mv/test.py::test_adding_replica 2.28s call test_storage_iceberg/test.py::test_evolved_schema_complex[azure-2] 2.27s call test_storage_iceberg/test.py::test_cluster_table_function_with_partition_pruning[s3-2] 1.96s call test_storage_iceberg/test.py::test_evolved_schema_complex[s3-2] 1.53s teardown test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table 1.51s call test_s3_cluster/test.py::test_ambiguous_join 1.49s call test_storage_iceberg/test.py::test_evolved_schema_complex[s3-1] 1.13s call test_replicated_user_defined_functions/test.py::test_drop_if_exists 1.03s call test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[true-expected_raise0] 1.01s teardown test_shutdown_static_destructor_failure/test.py::test_shutdown 0.93s call test_settings_randomization/test.py::test_settings_randomization 0.79s call test_refreshable_mv/test.py::test_pause 0.75s call test_replicated_user_defined_functions/test.py::test_create_and_drop 0.69s call test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 0.65s call test_s3_cluster/test.py::test_wrong_cluster 0.60s call test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[false-expected_raise1] 0.59s call test_s3_cluster/test.py::test_unset_skip_unavailable_shards 0.38s call test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage 0.37s call test_s3_cluster/test.py::test_skip_unavailable_shards 0.34s call test_render_log_file_name_templates/test.py::test_check_file_names 0.33s call test_ssh/test.py::test_simple_query_with_paramiko 0.29s call test_ssh/test.py::test_create_table 0.27s call test_shard_level_const_function/test.py::test_remote 0.24s call test_ssh/test.py::test_simple_query_with_openssh_client 0.04s teardown test_storage_iceberg/test.py::test_cluster_table_function[s3-1] 0.02s call test_ssh/test.py::test_no_inserts_from_infile 0.02s call test_ssh/test.py::test_no_selects_into_outfile 0.02s call test_ssh/test.py::test_no_queries_from_file 0.02s teardown test_storage_iceberg/test.py::test_cluster_table_function[azure-1] 0.01s teardown test_replicated_user_defined_functions/test.py::test_reload_zookeeper 0.01s teardown test_storage_iceberg/test.py::test_delete_files[local-1] 0.01s teardown test_storage_iceberg/test.py::test_cluster_table_function[azure-2] 0.01s setup test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys 0.01s teardown test_recompression_ttl/test.py::test_recompression_multiple_ttls 0.01s teardown test_storage_iceberg/test.py::test_cluster_table_function[s3-2] 0.01s setup test_storage_iceberg/test.py::test_cluster_table_function_with_partition_pruning[s3-2] 0.01s teardown test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys 0.00s teardown test_settings_constraints_distributed/test.py::test_insert_clamps_settings 0.00s teardown test_storage_iceberg/test.py::test_delete_files[azure-1] 0.00s setup test_refreshable_mv/test.py::test_backup_outer_table 0.00s setup test_recompression_ttl/test.py::test_recompression_replicated 0.00s teardown test_shard_names/test.py::test_incorrect_shard_names 0.00s setup test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5] 0.00s teardown test_storage_iceberg/test.py::test_delete_files[azure-2] 0.00s setup test_refreshable_mv/test.py::test_pause 0.00s setup test_storage_iceberg/test.py::test_cluster_table_function[s3-2] 0.00s setup test_storage_iceberg/test.py::test_cluster_table_function[s3-1] 0.00s setup test_storage_iceberg/test.py::test_cluster_table_function[azure-2] 0.00s setup test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3] 0.00s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4] 0.00s setup test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] 0.00s setup test_replicated_user_defined_functions/test.py::test_start_without_zookeeper 0.00s setup test_replicated_user_defined_functions/test.py::test_replication 0.00s setup test_refreshable_mv/test.py::test_backup_inner_table 0.00s teardown test_replicated_user_defined_functions/test.py::test_replication 0.00s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2] 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-2] 0.00s setup test_storage_iceberg/test.py::test_delete_files[local-1] 0.00s setup test_replicated_user_defined_functions/test.py::test_replication_replace_by_another_node_after_creation 0.00s setup test_storage_iceberg/test.py::test_delete_files[azure-2] 0.00s teardown test_storage_iceberg/test.py::test_delete_files[s3-1] 0.00s teardown test_recompression_ttl/test.py::test_recompression_replicated 0.00s setup test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore 0.00s setup test_shard_names/test.py::test_shard_names 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_complex[azure-1] 0.00s setup test_s3_cluster/test.py::test_cluster_with_header 0.00s teardown test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke 0.00s setup test_recompression_ttl/test.py::test_recompression_simple 0.00s setup test_storage_iceberg/test.py::test_delete_files[s3-2] 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-1] 0.00s teardown test_s3_cluster/test.py::test_cluster_format_detection 0.00s setup test_s3_cluster/test.py::test_skip_unavailable_shards 0.00s setup test_refreshable_mv/test.py::test_refreshable_mv_in_read_only_node 0.00s teardown test_replicated_user_defined_functions/test.py::test_server_restart 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-2] 0.00s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3] 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-1] 0.00s teardown test_storage_iceberg/test.py::test_delete_files[local-2] 0.00s teardown test_replicated_user_defined_functions/test.py::test_create_and_replace 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-1] 0.00s setup test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] 0.00s setup test_refreshable_mv/test.py::test_refreshable_mv_in_system_db 0.00s setup test_storage_iceberg/test.py::test_delete_files[local-2] 0.00s teardown test_s3_cluster/test.py::test_select_all 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_complex[s3-1] 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_complex[local-1] 0.00s teardown test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-2] 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_complex[local-2] 0.00s teardown test_storage_iceberg/test.py::test_delete_files[s3-2] 0.00s teardown test_s3_cluster/test.py::test_ambiguous_join 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-1] 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-1] 0.00s setup test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6] 0.00s teardown test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-1] 0.00s setup test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] 0.00s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1] 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-2] 0.00s setup test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2] 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-1] 0.00s setup test_storage_iceberg/test.py::test_delete_files[s3-1] 0.00s setup test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4] 0.00s setup test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[true-expected_raise0] 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_complex[azure-2] 0.00s setup test_ssh/test.py::test_simple_query_with_openssh_client 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-2] 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_complex[local-2] 0.00s teardown test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5] 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-1] 0.00s setup test_s3_cluster/test.py::test_cluster_format_detection 0.00s setup test_s3_cluster/test.py::test_cluster_default_expression 0.00s setup test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[false-expected_raise1] 0.00s teardown test_replicated_user_defined_functions/test.py::test_create_and_drop 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-2] 0.00s teardown test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-2] 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_complex[azure-1] 0.00s teardown test_refreshable_mv/test.py::test_refreshable_mv_in_read_only_node 0.00s setup test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] 0.00s setup test_storage_iceberg/test.py::test_delete_files[azure-1] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_named_collection 0.00s setup test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_complex[local-1] 0.00s teardown test_s3_cluster/test.py::test_count 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_complex[azure-2] 0.00s setup test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 0.00s teardown test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[false-expected_raise1] 0.00s teardown test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[true-expected_raise0] 0.00s setup test_replicated_user_defined_functions/test.py::test_drop_if_exists 0.00s setup test_s3_cluster/test.py::test_count_macro 0.00s setup test_s3_cluster/test.py::test_select_all 0.00s teardown test_replicated_user_defined_functions/test.py::test_replication_replace_by_another_node_after_creation 0.00s setup test_replicated_user_defined_functions/test.py::test_create_and_replace 0.00s setup test_ssh/test.py::test_simple_query_with_paramiko 0.00s setup test_storage_iceberg/test.py::test_evolved_schema_complex[s3-2] 0.00s setup test_s3_cluster/test.py::test_count 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_complex[s3-2] 0.00s setup test_replicated_user_defined_functions/test.py::test_server_restart 0.00s teardown test_s3_cluster/test.py::test_cluster_with_header 0.00s teardown test_storage_iceberg/test.py::test_cluster_table_function_with_partition_pruning[s3-2] 0.00s teardown test_s3_cluster/test.py::test_count_macro 0.00s setup test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 0.00s setup test_s3_cluster/test.py::test_cluster_with_named_collection 0.00s teardown test_s3_cluster/test.py::test_cluster_default_expression 0.00s setup test_s3_cluster/test.py::test_union_all 0.00s teardown test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] 0.00s setup test_ssh/test.py::test_no_queries_from_file 0.00s teardown test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db 0.00s setup test_ssh/test.py::test_no_selects_into_outfile 0.00s setup test_replicated_user_defined_functions/test.py::test_reload_zookeeper 0.00s setup test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db 0.00s teardown test_s3_cluster/test.py::test_skip_unavailable_shards 0.00s setup test_ssh/test.py::test_no_inserts_from_infile 0.00s teardown test_storage_iceberg/test.py::test_evolved_schema_complex[s3-1] 0.00s teardown test_ssh/test.py::test_create_table 0.00s teardown test_ssh/test.py::test_simple_query_with_openssh_client 0.00s setup test_s3_cluster/test.py::test_unset_skip_unavailable_shards 0.00s teardown test_replicated_user_defined_functions/test.py::test_drop_if_exists 0.00s setup test_s3_cluster/test.py::test_wrong_cluster 0.00s teardown test_s3_cluster/test.py::test_union_all 0.00s teardown test_ssh/test.py::test_no_selects_into_outfile 0.00s teardown test_ssh/test.py::test_no_queries_from_file 0.00s teardown test_ssh/test.py::test_no_inserts_from_infile 0.00s teardown test_s3_cluster/test.py::test_unset_skip_unavailable_shards =========================== short test summary info ============================ FAILED test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys - As... FAILED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5] FAILED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated FAILED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference FAILED test_refreshable_mv/test.py::test_backup_inner_table - helpers.client.... FAILED test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1] FAILED test_secure_socket/test.py::test - assert (1793303874.5993814 - 179330... FAILED test_refreshable_mv/test.py::test_backup_outer_table - helpers.client.... FAILED test_refreshable_mv/test.py::test_pause - helpers.client.QueryRuntimeE... PASSED test_ssh/test.py::test_create_table PASSED test_ssh/test.py::test_no_inserts_from_infile PASSED test_ssh/test.py::test_no_queries_from_file PASSED test_ssh/test.py::test_no_selects_into_outfile PASSED test_ssh/test.py::test_simple_query_with_openssh_client PASSED test_ssh/test.py::test_simple_query_with_paramiko PASSED test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] PASSED test_replicated_user_defined_functions/test.py::test_create_and_drop PASSED test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[false-expected_raise1] PASSED test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1] PASSED test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[true-expected_raise0] PASSED test_s3_cluster/test.py::test_ambiguous_join PASSED test_replicated_user_defined_functions/test.py::test_create_and_replace PASSED test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] PASSED test_replicated_user_defined_functions/test.py::test_drop_if_exists PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2] PASSED test_refreshable_mv/test.py::test_adding_replica PASSED test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys PASSED test_s3_cluster/test.py::test_cluster_default_expression PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3] PASSED test_shard_names/test.py::test_incorrect_shard_names PASSED test_s3_cluster/test.py::test_cluster_format_detection PASSED test_shard_names/test.py::test_shard_names PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4] PASSED test_s3_cluster/test.py::test_cluster_with_header PASSED test_s3_cluster/test.py::test_cluster_with_named_collection PASSED test_s3_cluster/test.py::test_count PASSED test_storage_iceberg/test.py::test_cluster_table_function[azure-1] PASSED test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6] PASSED test_s3_cluster/test.py::test_count_macro PASSED test_s3_cluster/test.py::test_select_all PASSED test_s3_cluster/test.py::test_skip_unavailable_shards PASSED test_recompression_ttl/test.py::test_recompression_multiple_ttls PASSED test_s3_cluster/test.py::test_union_all PASSED test_s3_cluster/test.py::test_unset_skip_unavailable_shards PASSED test_s3_cluster/test.py::test_wrong_cluster PASSED test_storage_iceberg/test.py::test_cluster_table_function[azure-2] PASSED test_recompression_ttl/test.py::test_recompression_replicated PASSED test_storage_iceberg/test.py::test_cluster_table_function[s3-1] PASSED test_replicated_user_defined_functions/test.py::test_reload_zookeeper PASSED test_replicated_engine_arguments/test.py::test_replicated_engine_with_arguments PASSED test_remove_stale_moving_parts/test.py::test_remove_stale_moving_parts_without_zookeeper PASSED test_storage_iceberg/test.py::test_cluster_table_function[s3-2] PASSED test_replicated_user_defined_functions/test.py::test_replication PASSED test_settings_constraints_distributed/test.py::test_insert_clamps_settings PASSED test_storage_iceberg/test.py::test_cluster_table_function_with_partition_pruning[s3-2] PASSED test_replicated_user_defined_functions/test.py::test_replication_replace_by_another_node_after_creation PASSED test_recompression_ttl/test.py::test_recompression_simple PASSED test_storage_iceberg/test.py::test_delete_files[azure-1] PASSED test_render_log_file_name_templates/test.py::test_check_file_names PASSED test_storage_iceberg/test.py::test_delete_files[azure-2] PASSED test_replicated_user_defined_functions/test.py::test_server_restart PASSED test_replicated_database_alter_modify_order_by/test.py::test_alter_modify_order_by PASSED test_settings_constraints_distributed/test.py::test_select_clamps_settings PASSED test_settings_randomization/test.py::test_settings_randomization PASSED test_storage_iceberg/test.py::test_delete_files[local-1] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] PASSED test_storage_iceberg/test.py::test_delete_files[local-2] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] PASSED test_storage_iceberg/test.py::test_delete_files[s3-1] PASSED test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable PASSED test_storage_iceberg/test.py::test_delete_files[s3-2] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[azure-1] PASSED test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[azure-2] PASSED test_s3_zero_copy_ttl/test.py::test_ttl_move_and_s3 PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[local-1] PASSED test_replicated_user_defined_functions/test.py::test_start_without_zookeeper PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[local-2] PASSED test_storage_delta/test_imds.py::test_session_token PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[s3-1] PASSED test_storage_iceberg/test.py::test_evolved_schema_complex[s3-2] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-1] PASSED test_shutdown_static_destructor_failure/test.py::test_shutdown PASSED test_shard_level_const_function/test.py::test_remote PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-2] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-1] PASSED test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-2] PASSED test_refreshable_mv/test.py::test_refreshable_mv_in_read_only_node PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-1] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-2] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-1] PASSED test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-2] PASSED test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db PASSED test_refreshable_mv/test.py::test_refreshable_mv_in_system_db PASSED test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss SKIPPED [1] test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py:29: Disabled for debug and sanitizers. Too slow. ============= 9 failed, 90 passed, 1 skipped in 920.36s (0:15:20) ============== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_sbkw4t --privileged --dns-search='.' --memory=30703235072 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel2_0.jsonl --report-log-exclude-logs-on-passed-tests 'test_parallel_replicas_over_distributed/test.py::test_parallel_replicas_over_distributed[test_single_shard_multiple_replicas-4-1]' test_recompression_ttl/test.py::test_recompression_multiple_ttls test_recompression_ttl/test.py::test_recompression_replicated test_recompression_ttl/test.py::test_recompression_simple test_refreshable_mv/test.py::test_adding_replica test_refreshable_mv/test.py::test_backup_inner_table test_refreshable_mv/test.py::test_backup_outer_table test_refreshable_mv/test.py::test_pause test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_refreshable_mv/test.py::test_refreshable_mv_in_read_only_node test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db test_refreshable_mv/test.py::test_refreshable_mv_in_system_db test_remove_stale_moving_parts/test.py::test_remove_stale_moving_parts_without_zookeeper test_render_log_file_name_templates/test.py::test_check_file_names test_replicated_database_alter_modify_order_by/test.py::test_alter_modify_order_by test_replicated_engine_arguments/test.py::test_replicated_engine_with_arguments test_replicated_merge_tree_compatibility/test.py::test_replicated_merge_tree_defaults_compatibility test_replicated_merge_tree_encrypted_disk/test.py::test_different_keys test_replicated_merge_tree_encrypted_disk/test.py::test_same_keys test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table 'test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16]' 'test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12]' test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore test_replicated_user_defined_functions/test.py::test_create_and_drop 'test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[false-expected_raise1]' 'test_replicated_user_defined_functions/test.py::test_create_and_drop_udf_on_cluster[true-expected_raise0]' test_replicated_user_defined_functions/test.py::test_create_and_replace test_replicated_user_defined_functions/test.py::test_drop_if_exists test_replicated_user_defined_functions/test.py::test_reload_zookeeper test_replicated_user_defined_functions/test.py::test_replication test_replicated_user_defined_functions/test.py::test_replication_replace_by_another_node_after_creation test_replicated_user_defined_functions/test.py::test_server_restart test_replicated_user_defined_functions/test.py::test_start_without_zookeeper test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 'test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header]' test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster test_s3_zero_copy_ttl/test.py::test_ttl_move_and_s3 test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage test_secure_socket/test.py::test 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case1]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case2]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case3]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case4]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case5]' 'test_send_request_to_leader_replica/test.py::test_alter_table_drop_partition[case6]' test_settings_constraints_distributed/test.py::test_insert_clamps_settings test_settings_constraints_distributed/test.py::test_select_clamps_settings test_settings_randomization/test.py::test_settings_randomization test_shard_level_const_function/test.py::test_remote test_shard_names/test.py::test_incorrect_shard_names test_shard_names/test.py::test_shard_names test_shutdown_static_destructor_failure/test.py::test_shutdown test_ssh/test.py::test_create_table test_ssh/test.py::test_no_inserts_from_infile test_ssh/test.py::test_no_queries_from_file test_ssh/test.py::test_no_selects_into_outfile test_ssh/test.py::test_simple_query_with_openssh_client test_ssh/test.py::test_simple_query_with_paramiko test_storage_delta/test_imds.py::test_session_token 'test_storage_iceberg/test.py::test_cluster_table_function[azure-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[azure-2]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-2]' 'test_storage_iceberg/test.py::test_cluster_table_function_with_partition_pruning[s3-2]' 'test_storage_iceberg/test.py::test_delete_files[azure-1]' 'test_storage_iceberg/test.py::test_delete_files[azure-2]' 'test_storage_iceberg/test.py::test_delete_files[local-1]' 'test_storage_iceberg/test.py::test_delete_files[local-2]' 'test_storage_iceberg/test.py::test_delete_files[s3-1]' 'test_storage_iceberg/test.py::test_delete_files[s3-2]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[azure-1]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[azure-2]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[local-1]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[local-2]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[s3-1]' 'test_storage_iceberg/test.py::test_evolved_schema_complex[s3-2]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-1]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-azure-2]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-1]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-local-2]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-1]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[False-s3-2]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-1]' 'test_storage_iceberg/test.py::test_evolved_schema_simple[True-azure-2]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 ' returned non-zero exit status 1.